Event Processing Course

Event Processing Course

Event processing networks(relates to chapter 6)

2

Lecture outline

• Event processing network and its components

• Event processing agents

• Event channels

• Global states

3

Event processing network

Event Producer 1

Event Consumer 1

Event Consumer 2

Event Producer 2

Event Consumer 3

Agent 2

Channel

Agent 1

State

Agent 3

Event Producer 1

Event Consumer 1

Agent 2Agent 1

Link in the graph

means – event (or

stream) flowing from

the output terminal

to the input terminal

Connection can

be established

either by the source

or by the sink

4

Formal definition

• EPN = (V, E) is a directed (possibly cyclic) graph

• V = {C, P, EC, GS, EPA} where

– C = consumers, P = producers, EC = event channels, GS = global stores, EPA = Event Processing Agents

• E = { (C, EC), (EC, EPA), (EC, P), (EPA, EC), (EPA, GS)}

5

Recursive EPN

A32

C31A31

A21

A22

A4

A2 A3

A4

A1

A1

A23

P31

P22

P21 C21

C22

C23 A33

6

EPN implementation perspective

Single run-time artifact

Conceptual EPN

Segment Iprocessing

Segment Nprocessing

. . .

Function IProcessing Function N

Processing . . .

Location Iprocessing

. . . Location Nprocessing

EPA/Context partition

7

Event Processing Agent

Filtering

Matching

Derivation

Event

Processing

Agent

8

EPA types

Event Processing Agent

Filter Transform Detect Pattern

Translate Aggregate Split Compose

Enrich Project

9

Filter EPA

A filter EPA is an EPA that performs filtering only, and has no matching or derivation steps, so it does not transform the input event.

Filtering

Filter EPA

Filtered Out

Filtered In

Non-Filterable

Principal filter expression

10

Formal definition of Filter operation

• Filter is a function that maps a single event ev to the set {TRUE, FALSE, UNDECIDED) relative to an assertion, such that:

– ev is mapped to TRUE, if ev satsifies the assertion

– ev is mapped to FALSE, if ev doesn’t satisfy the assertion

– ev is mapped to UNDECIDED, if is undecided whether ev satisfies the assertion

• The event ev is unchanged.

11

Transform EPA A transform EPA is an EPA that performs the derivation function, and optionally also the filtering function

Derivation

Output

Failure

Filtering

12

Formal definition of Transform operation

• Transform is a function EV1 Ev2 such that

– EV1 is a collection of input events

– Ev2 is a collection of output events

– The transform type and function determine the mapping from EP1 to EP2

13

Transform EPA sub types

Translate Compose

Aggregate Enrich

Split

Project

14

Transform EPA subtypes definitions

• A translate EPA is a stateless transform EPA that takes as an input a single event, and generates a single derived event which is a function of the input event, using a translation formula.

• An enrich EPA is a translate EPA that takes a single input event, matches it against a global state element, and creates a derived event which includes the original event, with possible modified attributes, and an additional collection of attributes {A1,…,An} copied or calculated as a result of using the global state.

• A project EPA is a subtype of the translate EPA that takes an input event, and creates a single derived event that contains a subset of the attributes of the input event

15

Transform EPA subtype definitions (cont.)

• An aggregate EPA is a transform EPA that takes as input a collection of events and creates a single derived event by applying an aggregation function over the input events.

• A split EPA is a transform EPA that takes as an input a single event and creates a collection of events, each of them can be a clone of the original event, or a projection of that event containing a subset of its attributes

• A compose EPA is a transform EPA that takes groups of events from different input terminals, looks for matches using some matching criterion and then creates derived events based on these matched events.

Pattern detection EPAs are discussed

In Lecture 6

16

Pattern detection EPA

• A pattern detection EPA is an EPA that performs a pattern matching function on one or more input streams. It emits one or more derived events if it detects an occurrence of the specified pattern in the input events

Pattern detection EPAs are discussed

In Lecture 7

17

EPA Definition Elements

Definition Element describingevent processing agent

Icon:

EPA IdentifierEPA type

Filter expression

EPA General information

Filtering Function

Pattern signature

Derivation formulae

Matching Function

Derivation Function:

IdentifierEvent typesFiltersSources

Input Terminals

IdentifierEvent typesFiltersSources

IdentifierEvent typesTargets

Output Terminals

IdentifierEvent typesTargets

Relationships

Global State referenceGlobal State reference

18

FFD EPN – The bid request system

Store reference

Store Request enrichment

Bid Request creator Bid Request

channel

Driverstatus

19

FFD EPN – The location identification

GPS sensor

Location service

Driverstatus

Neighborhoods

20

FFD EPN – The assignment system

Automatic assignment

Driver

Bid routing

Store reference

Manual assignment preparationBid Request

channelBid enrichment

Assignment manager

Alerts channel

Assignment channel

Assignment request channel

21

FFD EPN – The control system

Driver

Assignment request channel

Store

Delivery alert

Pick-up alert

Alerts channel

Assignment channel

Assignment not done

22

FFD EPN – The ranking and reporting system

Daily statistics creator

Ranking increase

Assignment channel Drivers’

Guild

Daily assignments calculator

Alerts channel

Ranking decrease

Improving note

DriverStatus

Permanent weak driver

Idle driver

Improving driver

Consistent strong driver

Consistent weak driver

Driver

23

Event Channel

An event channel is a processing element that receives

events from one or more source processing elements,

makes routing decisions, and sends the input events unchanged

to one or more target processing elements in accordance with these routing

decisions.

24

Event channels definition element

Definition Element describingEvent Channel

Icon:

Event channel identifier

Routing scheme typeRouting expressions

General information:

Routing Scheme

SecurityReliability and AvailabilityPerformance commitmentsOrdering requirements

Quality of Service Assertions

IdentifierEvent typesTargets

Output Terminals:

IdentifierEvent typesFiltersSources

Input Terminals:

IdentifierEvent typesFiltersSources

IdentifierEvent typesTargets

25

Formal definition of channel’s routing

• Channel routing is a function that maps an event into To, where To is a collection of output terminals (without changing it).

26

Routing schemes

• Fixed: The channel routes every event that it receives on any input terminal to every output terminal. In cases where there are multiple output terminals this means that separate copies of each input event are transmitted on each output terminal

• Type-based: The channel makes routing decisions based on the event type of the event that is being routed.

• Content-based: The routing decision is based on the event's content, this can be phrased as decision trees or decision tables, and are based on the input event content, as well as context information.

27

Global State

• Historical events retained in an event store so that they can be processed at a later phase.

• Reference data that event processing agents can access, for example to enrich or cleanse event data. Reference data is not maintained by the event processing system, but can affect the output of the event processing.

• State of external entities that, like reference data, is not maintained by the event processing system but which can be used as part of processing events. Examples include: current state of a business process, an airport alert level, weather related states (sun, clouds, rain, snow…). This state could change as a result of a (raw or derived) event being processed by an event consumer.

• Event processing state (whether persisted to disk or held in shared memory) that is accessible across event processing agents, and can be updated by an event processing application. This is typically maintained by the event processing platform.

28

Global State Definition Element

Global state element identifierGlobal state element type

Definition Element describingGlobal State Element

Icon:

Schema State MachineVariable list

General information

Global State Element Meta-Data

IdentifierEvent typesFiltersSources

Input Terminals:

IdentifierEvent typesFiltersSources

29

Streambase – event flow:

30

EPN – Event Zero

31

Lecture Summary

• In this lecture we have discussed the following topics:

– The concept of event processing network

– EPA

– Channel

– Global state