Operational Intelligence with WSO2 BAM

Gaining  Opera+onal  Intelligence  �with  WSO2  

BAM  

Director,  Research,  WSO2  Inc.      Visi6ng  Faculty,  University  of  Moratuwa      Member,  Apache  SoEware  Founda6on  

   Research  Scien6st,  Lanka  SoEware  Founda6on  

Srinath  Perera  

About  WSO2  ๏  Global  enterprise,  founded  in  2005  

by  acknowledged  leaders  in  XML,  web  services    technologies,  standards    and  open  source  

๏  Provides  only  open  source  plaHorm-­‐as-­‐a-­‐service  for  private,  public  and  hybrid  cloud  deployments  

๏  All  WSO2  products  are  100%  open  source  and  released  under  the  Apache  License  Version  2.0.  

๏  Is  an  Ac+ve  Member  of  OASIS,  Cloud  Security  Alliance,  OSGi  Alliance,  AMQP  Working  Group,  OpenID  Founda+on  and  W3C.  

2  

๏  Driven  by  Innova+on  

๏  Launched  first  open  source  API  Management  solu+on  in  2012  

๏  Launched  App  Factory  in  2Q  2013  

๏  Launched  Enterprise  Store  and  first  open  source  Mobile  solu+on  in  4Q  2013  

3  

What  WSO2  delivers  

Business  Model  

4  

More  Informa+on  !  

๏  Include  links  to  product  downloads,  white  paper  downloads  ,  etc.  

5  

Outline  § Connected  Business  and  Big  data  analy+cs  § Big  Data  Technologies  from  WSO2  

§ BAM  –  Batch  analy+cs  § CEP  –  Real  +me  analy+cs  § Lambda  Architecture  to  combine  

§ From  your  business  to  insights  § Iden+fy  KPIs  § Calculate  KPIs  § Build  Dashboards    § Add  Drilldown  § Alerts  and  Ac+ons    

About  Connected  Business  webinar  series  

§  March 05:Introduction to the Connected Business

§  March 12: Enterprise Integration made easy with WSO2 ESB

§  .. §  April 30: Gaining

Operational Intelligence with WSO2 BAM

§  May 7: The WSO2 Advantage for a Connected Business

Adap+ve  Connected  Business  

Connected  Business  

Be  Adap+ve  ๏  Capture  business  ac+vity  (iden+fied  by  messages,  transac+on  execu+on,  and  data  state  changes)  and  store  data  points  for  future  analy+cs  

๏  Deliver  automated  no+fica+ons  to  stakeholders  and  systems  based  on  business  ac+vity,  stakeholder  accountability,  and  authority.    

๏  Automa+cally  adapt  business  process  execu+on  based  on  events  and  current  condi+ons  

Big  Picture  

Collec+ng  Data  ๏  Data  collected  at  sensors  and  sent  to  big  data  system  via  events  or  flat  files  

๏  Event  Streams:  we  name  the  events  by  its  content/  originator    

•  Get  data  through    – Point  to  Point  – Event  Bus  

•  E.g.  Data  bridge  – a  thrib  based  transport  we  did  that  do  about  400k  events/  sec  

Making  Sense  of  Data  ๏  To  know  (what  happened?)  

๏  Basic  analy+cs  +  visualiza+ons  (min,  max,  average,  histogram,  distribu+ons  …  )  

๏  Interac+ve  drill  down  

๏  To  explain  (why)  

๏  Data  mining,  classifica+ons,  building  models,  clustering        

๏  To  forecast    

๏  Neural  networks,  decision  models    

Dashboards  and  last  Mile  §  Presenting information

o  To end user o  To decision takers o  To scientist

§  Interactive exploration §  Sending alerts

http://www.flickr.com/photos/stevefaeembra/3604686097/

Big  Data  Architecture  

Data  Collec+on  

•  Can  receive  events  via  SOAP,  HTTP,  JMS,  ..  

•  WSO2  Events  is  highly  op+mized  version  (400K  events  TPS)  

•  Default  Agents  and  you  can  write  custom  

agents.    

Agent agent = new Agent(agentConfiguration); publisher = new AsyncDataPublisher(

"tcp://localhost:7612", .. ); StreamDefinition definition =

new StreamDefinition(STREAM_NAME, VERSION);

definition.addPayloadData("sid", STRING); ... publisher.addStreamDefinition(definition); ... Event event = new Event(); event.setPayloadData(eventData); publisher.publish(STREAM_NAME, VERSION, event);

Business  Ac+vity  Monitor  

BAM  Hive  Query  Find  how  much  +me  spent  in  each  cell.    

CREATE EXTERNAL TABLE IF NOT EXISTS PlayStream …

select sid,

ceiling((y+33000)*7/10000 + x/10000) as cell, count(sid)

from PlayStream

GROUP BY sid, ceiling((y+33000)*7/10000 + x/10000);

Complex  Event  Processor  

CEP  Query  define partition sidPrt by PlayStream.sid, LocBySecStream.sid

from PlayStream#window.timeBatch(1sec)

select sid, avg(x) as xMean, avg(y) as yMean, avg(z) as zMean

insert into LocBySecStream partition by sidPrt

from every e1 = LocBySecStream ->

e2 = LocBySecStream [e1.yMean + 10000 > yMean

or yMean + 10000 > e1.yMean]

within 2sec select e1.sid

insert into LongAdvStream partition by sidPrt ;

Calculate the mean location of each player

every second

Detect more than 10m run

Lambda  Architecture  

Extrac+ng  Opera+onal  Intelligence  from  your  

Business  

Key  Performance  Indicators  (KPIs)  ๏  To  monitor,  you  need  to  measure  

๏  KPIs  are  the  bojom  line    

๏  For  a  organiza+on  it  may  be  profit,  or  revenue    

๏  For  marke+ng,  it  may  be  LEADs  generated    

๏  They  are  oben  domain  specifics  

๏  For  each  KPI,  we  need  to  see  history,  find  rela+onships,  and  some+me  possible  future  trends.  

Many  Stake  Holders,  Many  Views  

Decision  Process  

Dashboard    

๏  Idea  is  to  get  the  “Overall  idea”  in  a  glance  

๏  Like  your  car  dashboard    

๏  Support  for  personaliza+on,  you  can  build  your  own  dashboard.      

๏  This  is  the  entry  point  for  Drill  down  and  ac+vity  monitoring.  

Drill  Down  

๏  Find  the  problem,  and  find  what  caused  it.  

๏  It  is  like  “Finding  CPU  bojleneck”  

๏  Idea  is  to  recursively  find  categories  which  are  responsible  for  the  problem  

๏  E.g.  See  year’s  sales,  then  drill  down  to  a  par+cular  year  to  see  per  month  data.    

๏  See  country  level  data,  and  drill  down  to  see  state  wide  data.    

๏  Ac+vity  monitoring  is  a  key  ac+vity  here      

Case  Study:  Sobware  Support  Services  

๏  KPIs  

๏  Number  of  open  Issues    

๏  Mean  and  Max  +me  to  close  a  issue    

๏  SLA  viola+ons    

๏  List  of  long  running  issues    

๏  Issue  by  last  update  +me  

๏  Data    

๏  Collect  data  when  issue  is  created  or  updated  or  closed.    

๏  E.g.  IssueChange(id,  ac+on,  +me,  userID)  

๏  Emit  ID  as  the  correla+onID  

From  data  to  KPI  ๏  Group  issues  by  ID  grouping  each  issue  into  one  ac+vity  (Ac+vity  Monitoring)  

๏  Calculate  KPIs  for  each  ac+vity  

define partition IssueParition IssueChange.id;

from IssueChange[type=“open”] as s

->IssueChange[“closed”] as e

select e.time-s.time ..

Aggregate  data  in  different  level  to  support  drill  down  

๏  hour,  weekly,  and  monthly  

๏  Ver+cal,  product,    

Aggregate  to  support  Drilldown  

Dimension 1 Dimension 2

Dashboard  (Build  with  UES)  

Click to drill down

Click to drill down

Click to see details about Issues

Drill  Down  

๏  Click  and  drill  down  issues  in  each  category    

๏  Can  select  a  issue  and  see  all  ac+vi+es  happened  within  the  issue  

๏  See  who  takes  most  +me  on  issues    

๏  Look  at  day,  week,  month  trends  

Analy+cs  Process  

Ques+ons?  

THANK  YOU!!!  

37