EVENT MANAGEMENT IN MULTIVARIATE STREAMING SENSOR DATA National and Kapodistrian University of Athens.

EVENT MANAGEMENT IN MULTIVARIATE STREAMING SENSOR DATA

National and Kapodistrian University of Athens

Event Management in Sensor Networkonline

offl

ine

What is an event?• The term “event” is used to describe an alteration on one

or more variables monitored by the system• Two kinds of processing modules with respect to an event

• Online event processing: focuses on real event detection, identification of time dependent correlations and causalities

• Offline event processing: event storage, post-processing of stored events and data -warehousing

Online event processing

0.8

0.6

0.5

Event

dete

ctio

n

Sensor streams

e1

e2

e3

e1e2

e2e3

E1 • 0• 1• 0• 0

E2 • 0• 1• 0• 0

En • 0• 0• 1• 0

0.2 150.0 0.12

250.0 8.0 248.3

23.0 21.4 342.1

0.15

251.0

22.9

s1

s2

s3

Event streams

Probabilistic Temporal Reasoning

Dependency structure

tt

e1 e2 e3

2 3 1e e e

Event correlation

Event

pre

dic

tion

1 2e e

2 3e eAdaptive

Filtering

Event/Change Detection• Sensor streams arrives as raw data that provide instant

measurements• Generation of event streams over an existing set of

sensor streams• The problem concerns both detecting whether or not a

change has occurred, or whether several changes might have occurred, and identifying the times of any such changes.

Event/Change detection algorithms• Change detection algorithms

• Cumulative Sum (CUSUM)• Shewhart Controller• Multivariate Autoregressive Model (MAR)

CUSUM(1/3)• The input parameters for the CUSUM algorithm are the

following:• the target value μ• the above-tolerance value • the below-tolerance value • the above-threshold value • the below-threshold value

• The output parameters for the CUSUM algorithm are the following:• the above-detection signal {0,1}• the below-detection signal {0,1}

CUSUM (2/3)

CUSUM (3/3)• Experiment set up

• μ = 0.5• = = 1.3

Uni

vari

ateti

me

seri

es x

t

(Acc

eler

ation

m/s

ec)

Time steps t

Positive change

Negative change

0.5, 0.3, 1.3k k thresh thresh

Time steps t

Cum

ulati

ve s

ums

Positive sum P

Negative sum N

0.5, 0.3, 1.3k k thresh thresh

Shewhart Controller (1/3)• In the Shewhart control chart, a variable is detected to

deviate at time t from its normality whenever exceeds one of the control limits

• Control limits• Upper Control Limit (UCL)

+

• Lower Control Limit (LCL)• = -

Shewhart Controller (2/3)

Shewhart Controller (3/3)

Univ

aria

tetim

e se

ries

x t(A

ccel

erati

on m

/sec

)

Time steps t

UCL and LCL

Detected change

3k

UCL

LCL

Multivariate Autoregressive (MAR)

Multivariate Autoregressive (MAR)

Time steps t

Time steps t

Varia

ble

2 es

timati

on

(Lum

inan

cecd

/m2)

1,tx 1,tx

2,tx 2,tx 2

2

Varia

ble

1 es

timati

on

(Lum

inan

cecd

/m2)

Relati

ve Err

or e 2

t

Time steps t

2,te 2thresh Detected change 2 7%thresh

Relati

ve Err

or e 1

t

Time steps t

1,te 1thresh Detected change 1 5%thresh

Event Correlation• Technique for making sense of a large number of events

and pinpointing the few events that are really important in that mass of information

• Accomplished by looking for and analyzing relationships between events.

• Implemented by a piece of software called “event correlator”

Event correlation: step-by-step• Event filtering

• consists in discarding events that are deemed to be irrelevant by the event correlator

• Event aggregation• a technique where multiple events that are very similar (but not

necessarily identical) are combined into an aggregate that represents the underlying event data

• Event masking• consists in ignoring events pertaining to systems that are

downstream of a failed system

• Root cause analysis•  It consists in analyzing dependencies between events, based for

instance on a model of the environment and dependency graphs, to detect whether some events can be explained by others

Event Correlation Engine (ECE)• Typical event correlation scheme (univariate data)

• A transition from object (i.e., event or sequence of events) A to object B occurs if and only if B occurs immediately after A (i.e., not within a time window).

• Only one object is considered at each step of the sequence (i.e., there are no objects occurring at the same time).

• Event correlation over multivariate sensor data• an alerting situation or a malfunctioning system is expected to lead

to several events triggered at the same time step.

Correlation of Multivariate Event Data• Stepwise correlation

• Based on a first order Markov chain

• Variable-order correlation of Multivariate Event Data• Based on idea of partial matching [Fan et al. 1999]

• Event correlation based on sliding window• Hybrid scheme that correlates events within a time window

Stepwise Correlation

1 1,0,1EV AC

3 0,0,0EV

= 1ACPA B C

1 0 1

0 1 1

0 0 0

1 0 1

0 1 0

0 1 0

0 0 0

1 0 0

0 1 0

BCP1

=3

BC

2 0,1,1EV

AC

ACP1

=3

AC,BCP = 1

P

1=3

BC,P = 1

BCP1

=2

BCAC

ACP1

=2

AC,BCP = 1

4 1,0,1EV BCP1

=4

BCAC

ACP2

=4

AC,BCP = 1

P

1=4

BC,P = 1

5 0,1,0EV BCP1

=5

BCAC

ACP2

=5

AC,BCP1

=2

P

1=5

BC,P = 1

BAC,BP

1=2

BP1

=5

6 0,1,0EV BCP1

=6

BCAC

ACP2

=6

AC,BCP1

=2

P

1=6

BC,P = 1

BAC,BP

1=2

BP2

=6

,BCP = 1

B,BP = 1

,BCP = 1

,BCP = 1

A B C

1 0 1

0 1 1

0 0 0

1 0 1

0 1 0

0 1 0

0 0 0

1 0 0

0 1 0

7 0,0,0EV BCP1

=7

BCAC

ACP2

=7

AC,BCP1

=2

P2

=7

BC,P = 1

BAC,BP

1=2

BP2

=7

B,BP1

=2

,BCP = 1B,P

1=2

8 1,0,0EV BCP1

=8

BCAC

ACP2

=8

AC,BCP1

=2

P2

=8

BC,P = 1

BAC,BP

1=2

BP2

=8

B,BP1

=2

,BCP

1=2B,P

1=2

AAP

1=8

,AP

1=2

9 0,1,0EV

BCP1

=9

BCAC

ACP2

=9

AC,BCP1

=2

P2

=9

BC,P = 1

BAC,BP

1=2

BP3

=9

B,BP1

=2

,BCP

1=2B,P

1=2

AAP

1=9

,AP1

=2

A,BP = 1

Variable-order correlation • Partial matching algorithm [Fan et al.199]

A B C

1 0 1

0 1 1

0 0 0

1 0 0

0 1 0

A/1 C/1

A/1

C/2 B/1 C/1 BC/1

B/1 C/1 BC/1

AC/1

AC/1

B/1 C/1 BC/1

B/1 BC/1

A/1

B/1 C/1 BC/1 B/1 C/1 BC/1B/1 C/1 BC/1

B/1 BC/1

1 1,0,1EV

/1

2, 1m l

AC/1C/2

2 0,1,1EV

3 0,0,0EV

/1

/1/1/1/1/1/1 /1 /1/1

/1

/1

Variable-order correlation

2, 1m l

A B C

1 0 1

0 1 1

0 0 0

1 0 0

0 1 0

A/2

B/1 C/1 BC/1 B/1 C/1 BC/1B/1 C/1 BC/1

B/1 BC/1

/1 AC/1C/2 /1

/1/1/1/1/1/1 /1 /1/1

/1A/1 A/1

/1

A/1

A/1

4 1,0,0EV

5 0,1,0EV A/2

B/2 C/1 BC/1

B/1 C/1 BC/1B/1 C/1 BC/1

B/2 BC/1

/1

AC/1C/2

/1

/1/1/1

/1/1/1

/1 /1/1

/1

A/1 A/1

/1

A/1

A/1

B/1

Sliding window algorithm• Address time dependencies among events within a specific

timeframe• At each the algorithm the algorithm recalculates probability

values with respect to a sliding window taking into account the new event vector arrived at the current time step t

• The algorithm has memory of exactly w time steps• Directed graph G=(V, E) where V=P(I) is the power set of I={}

• Graph Vertexes :• Weighted transition edge:

Sliding window algorithm• Frequency of each vertex, a – indicator

• For estimating the probabilities within two nodes, b - indicator• The b-indicator examines whether the event sets of two nodes

occur at two, possibly separate, time steps .

Sliding window algorithm• Two steps • First step: t < w

• Frequency of each vertex-node• Probability of occurrence • Frequency of vV within the occurrence of some node u V

• Conditional probability

Sliding window algorithm• Second step: t > w

• Frequency of each vertex-node within the last w time

• Probability of occurrence • Frequency of vV within the last w after the occurrence of some

node u V

• Conditional probability:

Sliding window algorithm

A B C

1 0 1

0 1 0

1 0 0

1 1 0

3w 1 1,0,1EV

AC

CA

1,wAP = 1 1,w

CP = 11,wC,AP = 1

1,wA,CP = 1

1,wC,ACP = 1

1,wAC,CP = 1

1,wA,ACP = 1

1,wAC,AP = 1

2 0,1,0EV

AC

CA

2,wAP

1=2

2,wCP

1=2

2,wC,AP

1=2

2,wA,CP

1=2

2,wC,ACP

1=2

2,wAC,CP

1=2

2,wA,ACP

1=2

2,wAC,AP

1=2

B2,wC,BP

1=2

2,wBP

1=2

1,wACP = 1 2,w

ACP1

=2

2,wA,BP

1=2

2,wAC,BP

1=2

3 1,0,0EV

AC

CA

3,wAP

2=3

3,wCP

1=3

3,wC,AP

2=3

3,wA,CP

1=4

3,wC,ACP

1=33,w

AC,CP1

=3

3,wA,ACP

1=4

3,wAC,AP

2=3

B3,wC,BP

1=3

3,wBP

1=3

3,wACP

1=3

3,wA,BP

1=4

3,wAC,BP

1=3

3,wB,AP

1=2

3,wA,AP

1=2

4 1,1,0EV

A

4,wAP

1=3

B

4,wBP

2=3

4,wA,BP

1=2

4,wB,AP

1=4

4,wA,AP = 1

4,wB,BP

1=2

AB4,wB,ABP

1=2

4,wAB,BP = 1

4,wA,ABP = 1

4,wAB,AP = 1

4,wABP

1=3

Event processing• A method of tracking and analyzing (processing) streams

of information (data) about things that happen (events),

and deriving a conclusion from them• Complex event processing, or CEP, is event processing

that combines data from multiple sources to infer events or patterns that suggest more complicated circumstances

• Techniques for CEP• Event-pattern detection• Event abstraction• Event filtering• Event aggregation and transformation• Modeling event hierarchies

CEP categories

• Two main categories• Aggregation-oriented CEP: an aggregation-oriented CEP solution

is focused on executing on-line algorithms as a response to event data entering the system. A simple example is to continuously calculate an average based on data in the inbound events

• Detection-oriented CEP: focused on detecting combinations of events called events patterns or situations. A simple example of detecting a situation is to look for a specific sequence of events.

Adaptive filtering of rules• Use of aging or decay function • Linear or exponential degradation f t

tr f t

21 1

1i

kr i k

n

expir ki

1k

0.8k

0.3k

0.1k

i

ir

0.03k 0.06k 0.1k 0.3k

i

ir

,1

,

1

t

i r ii t t

r t t

ii t t

r pp

r

Rules probability