Solving Manufacturing Equipment Monitoring Through Efficient Complex Event Processing Tilmann Rabl, Kaiwen Zhang, Mohammad Sadoghi, Navneet Kumar Pandey,

Solving Manufacturing EquipmentMonitoring Through Efficient

Complex Event Processing

Tilmann Rabl, Kaiwen Zhang, Mohammad Sadoghi, Navneet Kumar Pandey, Aakash Nigam, Chen Wang, Hans-Arno Jacobsen

Middleware Systems Research Group, University of Toronto

DEBS Grand Challenge 2012

msrg.org2

Agenda

Complex Event Processing Scenarios

System Architecture

Evaluation

Demo

18/07/2012

msrg.org3

Motivation Course “Large-Scale Data Management”

Big data storage Large scale event processing

Complex event processing scenarios Application performance management Smart traffic monitoring Energy monitoring

Resulting team project DEBS Grand Challenge

18/07/2012

msrg.org4

Scenario I: Application Performance Management

Monitoring of enterprise systems Find bottlenecks, problems Trace transactions, measure utilization

18/07/2012

msrg.org5

Scenario II: Smart Traffic Monitoring

Traffic data from cars, mobile devices, road sensors

Event aggregation, filtering, correlation Traffic status, accident detection, etc.

18/07/2012

msrg.org6

Scenario III: Energy Monitoring

Green computing Application-level energy monitoring API-based energy consumption estimation

Operating System & Hardware

Applications

CPUNetwor

kI/O

Formulae

Sensors

…

Store

API

18/07/2012

msrg.org7

Common Denominators High data rates

1000 – 1000000+ events / sec Small data points

< 1 KB Complex queries

Filtering, aggregation, correlation Persistent storage Distributed setup

DEBS Grand Challenge

18/07/2012

msrg.org8

Continuous Query

Evaluation

High-Level Architecture Monitoring Service

Input data stream, marshalling Event Dissemination Substrate

(Optional) pub/sub layer, queues Continuous Query Evaluation

Consumes input, computes results Storage Manager

Stores data, enables querying Client

Visualizes query results Java-based implementation

Monitoring Service

Storage Manager

Client

Event Disseminati

on Substrate

Stable Storag

e

18/07/2012

msrg.org9

Storage Architecture Data is stored in

tables Key-value pairs

Table Index Fast lookup

Compressor Efficient data storage Run length encoding

Grand Challenge data Run-length: 20000 Compression: 99.99%

18/07/2012

msrg.org10

Client

Google Web Toolkit-based Client Java-code compiled to JavaScript Displays all Grand Challenge results

Plots, results, alarms

18/07/2012

msrg.org11

Evaluation Distributed setup

Separated servers for data generator and monitoring tool

Configuration 2 servers 2 x dual core Xeon processor 4 GB RAM Gigabit Ethernet

Data set: 5 min + 18 days + synthetic Synthetic data

Many errors (every 2 - 3 min) Maximum throughput (no network)

Metrics: latency & throughput18/07/2012

msrg.org12

Processing Overhead Real Workload

0 – 200 queries (Q1,Q2 repeated) Linear in the number of queries Stable with increasing generator speedup

18/07/2012

msrg.org13

ThroughputReal Workload

Throughput controlled by data generator Data generator does not saturate system Peak 9000 events/sec ~ 0.11 ms arrival rate Well below maximum throughput

18/07/2012

msrg.org14

LatencySynthetic Workload

Maximum throughput Minimum latency

0.019 ms (10 queries) Maximum latency

0.63 ms (1400 queries)

18/07/2012

msrg.org15

Throughput Synthetic Workload

Maximum throughput 40000 events / sec (20 queries)

Minimum throughput 1500 events / sec (1400 queries)

18/07/2012

msrg.org16

Demo

Live!

18/07/2012

msrg.org17

Conclusions Complex event processing scenarios

Application performance management Smart traffic monitoring Energy monitoring DEBS Grand Challenge

Efficient implementation of the Grand Challenge Java-based Google Web Toolkit GUI Synthetic data generator

Up to 40000 events per second Up to 1400 queries

18/07/2012

msrg.org18

Thank You!

Questions?

Contact: Tilmann Rabl tilmann.rabl@utoronto.ca msrg.org

18/07/2012

msrg.org19

Back-Up: DEBS Grand Challenge Manufacturing equipment monitoring Large, real data set

18 days, 100 Hz 2 queries

State of sensors and valves Difference Threshold

Power consumption Range Average Threshold

18/07/2012