Real-Time Processing Layer - ProaSense · Real-Time Processing Layer Goal-Driven Event Processing ... Panteon->Esper, Oracle->CQL) - Completely missing for distributed ... Pattern

FP7-ICT-2013.1.3

Brussels, 24.11.2014

Real-Time Processing LayerGoal-Driven Event Processing

Dominik Riemer, FZI

• Introduction– Introduction to Complex Event Processing

– Real-Time Processing Layer: Objectives

– Role of layer in the ProaSense architecture

– Research problems related to the layer

• Methodology: Overview

• Setup Phase

• Execution Phase

• Current Status and Roadmap– Current Status

– Future Work

Outline

2

INTRODUCTION

Complex Event ProcessingConceptual model

Sensors

Applications

Processes

Notifications

Actions

Even

t Pro

duce

rs

Even

t Con

sum

ersEvent Processing

Network

Relevant Situations

Humans

4

Real-Time Processing LayerChallenges and Objectives

ObjectivesChallenges

• The real-time processing layerhandles high-volume datastreams produced by a large variety of sensors

• Objective is to identify potential failures immediately

• many patterns are needed todetect such situations

• large-scale, distributed real-time infrastructure is needed

• Methodology for Goal-Driven Event Processing

• Models and implementation forsemantic descriptions ofheterogeneous event processingagents

• Modelling and deployment of real-time processing pipelines

5

Real-Time Processing LayerComponents in the ProaSense Architecture

6

Real-Time Processing LayerIdentified research problems

Pattern Complexity

Pattern Expressivity

Pattern Reusability

Pattern Identification

Design-time challenges of Complex Event Processing:How can non-technical users identify and model event patterns?

detailed next slides

Proactive monitoring of potential failures and situations of interest

restricted by

7

High complexity• complex structure for (relatively) simple use

cases• development effort similar to traditional

programming languages

Limited expressivity• many use cases typically require custom

event processing logic• limited expressivity of pattern languages

require implementation of external functions• e.g., detection of drilling start

Limited reusability• fixed event schemas• static definition of event properties• hard-coded values

Implementation example (Esper EPL)

Research ProblemsComplexity, Expressivity, ReusabilityTask: Detection of an abnormal oil temperature increase (e.g. 10% above normal) measured over 30% of the drilling period when drilling RPM exceeds a threshold …

INSERT INTO HighTemperatureEventSELECT * FROM pattern [a=DrillingProcessEvent ANDb=AggHighTemperatureEvent where timer:within(2 min)]

INSERT INTO DrillingProcessEventSELECT timestamp, variable_typeFROM DrillingRPM match_recognize(partition by deviceIdmeasures B.value as rpmpattern (A B) define A as A.drillingStart, B asB.drillingStop

create schema DrillingRPM as (variable_type string, timestamp long, value double, deviceId int)

create schema GearLubeOilTemperature as (variable_type string, timestamp long, value double)

INSERT INTO AggHighTemperatureEvent(g.variable_type, g.value, avg(g.value) SELECT * FROM GearLubeOilTemperature.win:time(2 min) as g group by variable_type

8

Identification

Scope of event patterns:• real-time/proactive monitoring• detection of threats• detection of opportunities

Requirements:• set of patterns is complete• set of patterns is error-free• support for heterogeneous technical systems

Problems:• gap between business requirements and

technical pattern representation

Research ProblemsPattern identificationTask: Identify all patterns which are of interest in a particular domain

PIPIPM5

PIPIP

ContextTechnical Representation

PIPIKPI

PIPIP

PIPIP

Business Requirements

9

What‘s wrong with existing approaches?Graphical tool support for CEP

- Programming-oriented developmentenvironments

- Definition of custom (e.g., JAVA-based) functions required

- Translation into exactly onelanguage (e.g., Panteon->Esper, Oracle->CQL)

- Completely missing for distributedstream processing

Example: Oracle CEPProblems

10

Semantic Event Processing Agent

Semantic Event Producer

Main contributionMethodology for Goal-Driven Event Processing

Methodology: Overview

Execution PhaseSetup Phase

Ontologies

Example

Conceptual Model

Ontologies

Example

Conceptual Model

Processing Pipelines

Conceptual Model

SemanticIntegration

Example

11

METHODOLOGY: OVERVIEW

Setup PhaseSetup Phase Execution PhaseExecution Phase

Pattern EngineersPattern Engineers

Business AnalystsBusiness Analysts

Technical ExpertsTechnical Experts

Methodology

Source Modelling

Stream Modelling

Stream Implementation

SEPA Modelling

Performance Indicators

Business Goals

Key Performance Indicators

Pipeline Implementation

Pipeline Deployment

Pipeline Evolution

Pipeline Identification

Register

SEPAOntologySEPA

Ontology

Requirements

Requirements


Event OntologyEvent

Ontology

SEPA Implementation

13

Concrete Implementation

Methodology: Outcome

High-level processing pipelines

Common semantic descriptionHow can heterogeneous event processing logic be integrated?

Concrete Implementation Concrete Implementation

StandaloneCEP Engine TopologyAlgorithm

Proa

Sens

eW

P3Im

plem

enta

tions

14

Semantic Event Producer (S-EP)Semantic Event Processing

Agent (S-EPA)Processing Pipeline

Increase

Aggregation

ShaftSpeed

Increase

Aggregation

ShaftSpeed

Concepts

• performs an eventprocessing logic

• provides input eventstream requirements, restrictions and outputdescription

• produces one or moreevent streams

• describes streams basedon functional and non-functional (e.g., quality) aspects

• transformation processfrom input event streamsto an output eventstream

• consists of a set of S-EPAs and S-EPs

• single pipeline elementscan be implemented byarbitrary underlyingsystems

• tree structure (consumeras top-level elements)

15

SETUP PHASE






Source Modeling Stream Modeling


SEPA Modeling

Business Goals

Strategy


Pipeline Deployment

Pipeline Evolution


Register

SEPAOntologySEPA

Ontology

Requirements

Requirements


Event OntologyEvent

Ontology

SEPA Implementation

Semantic Event Producer

17

Setup PhaseSemantic Event Producer: Ontologies

Semantic Event Producer Event Stream

Event Source

Event Quality

Event Grounding

Event Schema

Event Feature

rdf:Datatype

muo:Measurement Unit

non‐functional aspects:‐ e.g., frequency, latency, accuracy

source‐related aspects:‐ e.g., sensor/application/human

technical aspects:‐ e.g., transport protocol, data format

functional aspects:‐ e.g., event header, payload

produces

has..

eventFeature

measurementUnit

featureType

rdfs:classtypeOf

mhwirth:DDM mhwirth:DrillingSpeedShaft

Concept

Instance

mhwirth:speed

muo:rpm

xsd:doublexsd:string

value

featureName

18



Event Source

Event Quality

Event Grounding

Event Schema

Event Feature

rdf:Datatype






produces

has..

eventFeature

measurementUnit

featureType

rdfs:classtypeOf


Concept

Instance

mhwirth:speed

muo:rpm


value

featureName

Event Source• determines the source an event is

produced by• e.g., sensor, application, website• allows modelling of source-based

restrictions (e.g., static or movingsensors)

19



Event Source

Event Quality

Event Grounding

Event Schema

Event Feature

rdf:Datatype






produces

has..

eventFeature

measurementUnit

featureType

rdfs:classtypeOf


Concept

Instance

mhwirth:speed

muo:rpm


value

featureName

Event Quality• determines the quality level of a

stream• e.g., frequency, accuracy, certainty• linked to ssn:Quality

20



Event Source

Event Quality

Event Grounding

Event Schema

Event Feature

rdf:Datatype






produces

has..

eventFeature

measurementUnit

featureType

rdfs:classtypeOf


Concept

Instance

mhwirth:speed

muo:rpm


value

featureName

Event Grounding• determines the technical grounding

of an event stream (protocol, port, topic)

• e.g., JMS, MQTT, HTTP, Kafka

21



Event Source

Event Quality

Event Grounding

Event Schema

Event Feature

rdf:Datatype






produces

has..

eventFeature

measurementUnit

featureType

rdfs:classtypeOf


Concept

Instance

mhwirth:speed

muo:rpm


value

featureName

Event Schema• determines the (functional) schema

of a single event• Event Feature as single event

properties (e.g., sensor observations)

• features can have a measurement unit (MUO) and a specific type

22








SEPA Modeling

Business Goals

Strategy


Pipeline Deployment

Pipeline Evolution


Register

SEPAOntologySEPA

Ontology

Requirements

Requirements


Event OntologyEvent

Ontology

SEPA Implementation

Semantic Event Processing Agents

23

Setup PhaseSemantic Event Processing Agent: Ontologies

Event StreamSemantic Event Processing Agent

requires

Static Feature Output Strategy

outputStrategy

Append

Fixed

Custom

Rename

staticFeature

AnyStaticFeature

FreeTextStaticFeature

MappingStaticFeature

MatchingStaticFeature

OneOfStaticFeature

24



requires


outputStrategy

Append

Fixed

Custom

Rename

staticFeature

AnyStaticFeature




OneOfStaticFeature

Required event streams• restrictions on stream inputs

25


OneOfStaticFeature


requires


outputStrategy

Append

Fixed

Custom

Rename

staticFeature

AnyStaticFeature




OneOfStaticFeature

Output strategy• controls output of a SEPA• output depends on input schema(s)

and SEPA type• e.g., ENRICH SEPA type:

AppendOutputStrategy• currently supported: Append, Keep,

Replace, Fixed, Custom

26



requires


outputStrategy

Append

Fixed

Custom

Rename

staticFeature

AnyStaticFeature




OneOfStaticFeature

Static feature• defines static data required to

instantiate a SEPA• e.g., threshold values, time window

specifications,

27



requires


outputStrategy

Append

Fixed

Custom

Rename

staticFeature

AnyStaticFeature




OneOfStaticFeature

Mapping static feature• defines mapping of an event feature

to a SEPA operation

28



requires


outputStrategy

Append

Fixed

Custom

Rename

staticFeature

AnyStaticFeature




OneOfStaticFeature

Matching static feature• defines mapping between two event

features of different streams• e.g., A AND B where a.x=b.y

29

Setup PhaseSemantic Event Processing Agent: Example

Event Restrictions- payload contains numerical value- payload produced by a sensor

Quality Restrictions- min frequency 1 event/sec- min sensor accuracy

Static data requirements- time window- static threshold: value- dynamic threshold

Capabilities max throughtput

Event restrictions- events produced by speed shaft- measurement unit RPM

Quality restrictions- min frequency

Static data requirements- min RPM

Capabilities- at-least-once processing

Increase

Aggregation

Drilling Status

Algorithm

30

Setup PhaseSemantic Event Processing Agent: Example

Increase

Event Stream

EventFeature

xsd:number

featureType

eventFeature

requires

AnyStaticFeature

percentage

Option Option

staticFeature

option

absolute


value Drilling Status

Event Stream

EventFeature

muo:rpm

measurementUnit

eventFeature

requires

speedShaft

rdf:type

Increase

Aggregation

Drilling Status

Algorithm

31

EXECUTION PHASE








SEPA Modeling

Business Goals

Strategy


Pipeline Deployment

Pipeline Evolution


Register

SEPAOntologySEPA

Ontology

Requirements

Requirements


Event OntologyEvent

Ontology

SEPA Implementation

33

Processing PipelinesConceptual Model

ShaftSpeed

OilTemperature

Drilling Status

Algorithm

Drilling Status : Drilling

Filter

Increase

AggregationAND

Pattern Detect Dashboard

Event Consumer

abnormal oil temperature rise (10% above normal) measured during the drilling period

StandaloneCEP Engine

Topology

Algorithm

34

Semantic IntegrationExample

Connection valid?

Additional user input required?

Increase S‐EPA output schema?

?Increase

Aggregation

ShaftSpeed

35

Semantic Integration

IncreaseShaftSpeed

Define static properties and concrete mappings

Validate connectionCompute mapping properties

Compute output schema

Connection is validated based on four metrics:• Feature match, schema match, quality

match, grounding match

If connection is valid:• for all required mapping properties of P: find possible mappings in S

Task

Increase S-EPA requires an event stream with an event feature of type xsd:number Shaft Speed provides such a feature

(mhwirth:speed) -> connection valid

Find all event features in S and add mapping property list

Example

′ , ′ 1 | ∀ ∈ : ∃ ∈ ′ ∧0 |

Actor: System

36

Semantic IntegrationDefine static properties and

concrete mappingsValidate connection

Compute mapping propertiesCompute output schema

For all static properties of P• provide static properties as user‐definedinput (text fields, selections)

For all concrete mapping properties of P• provide list of mappings as user‐defined input (selection)

Task

Increase S-EPA requires a static property indicating a minimum increase threshold

As only mhwirth:speed matches the input requirements, no mapping must be chosen

Example

IncreaseShaftSpeed

Actor: User

37

Semantic IntegrationDefine static properties and concrete mappings

Validate connectionCompute mapping properties

Compute output schema

Compute output based on:

• concrete input event schema

• defined mapping properties

• predefined output strategy

Connect next element

For each S-EPA

• compute invocation graph

Task

Increase S-EPA defined output strategy Custom• users are able to manually define the

output

A Filter S-EPA would define an output strategy Keep, which is defined by the input.

Example

IncreaseShaftSpeed

Actor: System

38

Pipeline EditorPrototype

39

Real-Time Processing Layer - ProaSense · Real-Time Processing Layer Goal-Driven Event Processing ... Panteon->Esper, Oracle->CQL) - Completely missing for distributed ... Pattern

Documents