Part 1 A Simple Introduction to Complex Event Processing

The Power of Event

Part 1

A Simple Introduction to Complex Event Processing

Not For Me(@JoeWooJin)[email protected]

The Power of Event

Chapter 1

The Global Information Society and the Need for New Technology


Contents

● Events everywhere in our information systems● The Internet and the growth of global communication

spaghetti● Layer upon layers in enterprise system architecture● Global electronic trade - understanding what is

happening● Agile systems - future reality or just a dream● Can an open electronic society defend itself?● The gathering storm - global coordination or global

chaos

1.1 Distributed Information Systems Everywhere

● The Internet has promoted and speeded the growth of distributed information processing beyond the single enterprise, across the boundaries between enterprises.

● Financial Trading System, government and military information system

● underlying architectural strucuture is the same: a distributed information system, called "enterprise systems"

● business-level, or stragetic-level events; complex events.

1.2 The Global Communication Spaghetti Pot

● We can't tell where it starts and where it ends, we can't unravel it, and most of the time we don't know how it happend.

● The challenge is not to restrict communication flexibility, but to develop new technologies to understand it

● The technology of monitoring and managing events in IT systems has been completely overrun by the technology of communicating events.

● This is a global problem in search of new ideas on how to solve it.

1.2.1 Event Causality

● A key to understanding events is knowing what caused them.

● Events are flowing all the time through our Internet-based systems from one part of the world and causing events in another part of the world.

● Horizontal causality● Vertical causality

1.3 Electronic Archeology: Layers upon Layers

● Enterprise systems are distributed, event-driven

systems.

● They are layered systems.

● Layering is a design technique for controlling

complexity.

● Layered IT systems present another dimension in the

search for new ways to understand the events that

happen in them.

1.3.1 A Layered Enterprise System

● Corporate Level

● Applications and User at the Top; business level or

strategic planning level

● Collaboration and Enabling Layer

● Middleware

● Network Layer

1.3.2 Vertical Causality: Tracking Events up and down the Layers

● Acitivity at each layer is translated into activities at the layers below and conversely.

● An activitiy at the top causes activities at successively lower levels, which in turn cause other activities to happen at the top.

● An ability to track vertical causality in a layered systema. understanding properties of that high-level event,

such as its timing and how it improve performance at the business level.

b. to group events at lower levels according to the high-level events they signify.

1.3.3 Event Aggregation: Making High-Level sense out of Low Level Events

● The complementary operation to downward tracking of vertical causality is the aggregation of sets or groups of lower events into a single higher-level event that expresses the meaning of the lower-level events, taken together.

● Recognizing or detecting a significant group of lower-level events from among all the enterprise event traffic, and creating a single event that summarizes in its data their significance, is called event aggregation.

1.4 The Gathering Storm of New Activities on the Web

● Widely distributed fragments of information must be gathered and pieced together into forms appropriate for elctronic processes to tackle the problems at hand, and for humans to understand high-speed business situations and stay in control.

● We must be able to deal with increasingly complex systems, demand for high reliablity, and frequent, rapid modifications.a. Global Electronic Tradeb. Agile Systemsc. Cyber Warfare and the Open Electronic Society

The Power of Event

Chapter 2

Managing the Electronic Enterprise in the Global Event Cloud


Contents

● The global event cloud● How enterprises operate in the global event cloud● Management process - going beyond workflow● Autonomous parallel, asynchronous process● The electronic enterprise● Treating the exceptional situation as normal● Enabling the human to control the electronic enterprise● Technology demands for managing the electronic

enterprise

2.1 How the Global Event Cloud Forms

● The Open Enterprise○ Their boundaries become blurred.○ Events are flowing between the enterprise and its

trading partners, the electronic market hubs...● The Global Event Cloud

○ a "cloud" of events rather than a "stream" because the event traffic is not, in most cases, nicely organized.

● The Electronic Enterprise○ "Event driven" - those tools and applications rely on

receiving events to monitor he progress of a process and issuing eventss to initiate its next stage.

2.2 Operating in the Global Event Cloud 1/2

● Enterprise management processes today typically consist of a large number of linear workflows like this one, loosely strung together and nested one inside another

● ProcessOrder is a process for handling incoming orders from customers in our typical elctronic enterprise.

● It is an electronic middleman, perhaps using eMarketplaces to select its vendors.

ProcessOrder

1. New Order

2. Select Vendor

3. Pick, Pack, and Ship Product

4. Billing

Spare Part Query

Terminate Instance

Select Product

Ship Reqeust

GL QueryInvoice Print

Ship Quote

Ship Order

Invoice PrintSerice

Requeset

Valid ServiceProcess Order


1. New Order

2. Select Vendor

3. Pick, Pack, and Ship Product

4. Billing

Process ActivitiesValid Order When a valid order is

received, select the vendor with the lowest price.

When a vendor is selected, send a ship order to the vendor's warehouse.

When a UPS routing number is received, terminates the process.

Process RulesGlobal Event Cloud

Select Vendor

Vendor Selected

Ship Order

Product Shipped

Terminate Instance


● In Figure the ProcessOrder workflow is on the left, the global event cloud in the center, and the rules that drive the workflow process are on the right.

● The rules that drive the process are "reactive rules".● Each rule contains an event pattern called a trigger.● These events that drive the workflow process are

transported back and forth between the activities and the workflow rules engine by the enterprise IT layer.

● They are part of event cloud.

2.3 Going Beyond Workflow

● To enter the world of global Internet marketplace decision making, business and management process must meet the reduced time scales and increased situational complexities that will be involved.

● They soon will be parallel, asynchronous processes.● "Autonomous parallel processing"

○ Be completely automated and event driven.○ Execute in parallel.○ Make decision and communicate asynchronously.

2.4 Parallel and Asynchronous Process

● Scalable complex event pattern matching, taking into account the context or state at the time of a match

● The ability to reuse the data in sets of events that match a rules pattern in the creating of new events.

● These technology demands are the foundation for implementing parallel, asynchronous real-time management processes.

2.5 On-the-fly Process Evolution

● On-the-fly evolution means the ability to modify a process without halting the rules engines or disrupting the execution of other processes

● Technology demands for controlling and modifying electronic processes are○ Real-time, personalized viewing of activity at every

level in the enterprise.○ On-the-fly process modification○ Simulation of processes before going live

2.6 Exceptions Must Be First-Class Citizens in Process Design

● If a process fails to behave in a given situation as specified or meets a situation for which it has no specification, it is said to have encountered an exception.

● So, the process design technology should let us design normal processing and exceptional processing in the same way.

● However, there are some distinguishing problems in dealing with exceptions○ We must be made aware of their presence in real

time.○ We must be able to find out what causes them

The Power of Event

Chapter 3

Viewing the Electronic Enterprise -Keeping the Human in Control


Contents

● Event monitoring - the standard technology today● Enterprise viewing - a step beyond monitoring● Recognizing sets of events from the global event cloud -

a key to personalized viewing● Information gaps● Enterprise structure and abstraction hierarchies● Hierarchical viewing - the key to human control of the

enterprise

3.1 Today's Event Monitoring Is Too Primitive

● System Monitoring Focuses on the network layer.● Network-level monitoring doesn't even solve network

problems.● The tools contains very littel "smarts" to tell a manager

what the problem is and what to do.● They are faced daily with the following kinds of issues.

○ The network event logs can become very large and difficult to handle in real time.

○ Tools to aid in picking out sets of related events are needed.

○ Causal tracking is needed.○ Predictive monitoring is beyond the state of the art.

3.3 Information Gaps● Different people engaged in the operations of an enterprise need

different kinds of information.● This leads to information gaps between the kind of information

people need to do their jobs effectively and easily, and the information they actually get.

● An information gap generally has two dimensions:○ A vertical dimension, which is the difference between the level

of the enterprise at which events and other data are monitored and the level at which the user is operating within the enterprise

○ A horizontal dimension, which is the amount of analysis needed to render the monitored information in a userful form for the user's tasks/

3.4 Problem-Relevant Information

● To bridge information gaps we need a technology for constructing problem-relevant information from whatever events we can monitor.

● How to get problem-relevant information1. Relevance tothe problem of immediate interest2. Ease of understandability3. Ease of analysis4. Ease with which multiple views can be coordinated

3.5 Viewing Enterprise Systems 1/2

● A view of a system is a selection of information about what the system is doing currently or did in the past that is processed to abstract or extract those aspects relevant to a problem of interest.

● Examples○ An application-level activity view○ A who's-talking-to-whom view○ A network problems view○ A high-level system performance view○ A corporate business process tracking view

3.5 Viewing Enterprise Systems 2/2

● Each of these examples of a view has the following elements○ Each view has a problem of interest.○ Each view is event driven.○ The view are provided in humanly understandable

forms using graphics○ Each view provides relevant events that can be used

to drive automated decision making processes.○ Most important, a view must be easy to modify, on

the fly, to incorporate new types of events, change the aggregation technique

3.6 Creating and Coordinating Multiple Views

● Different people need different views.● Simply, this is because different users are interested in

different kinds of information about the system.● Not only do we need multiple views of a system, but

each user needs to be able to customize their own view.

3.7 Hierarchical Viewing

● A powerful technique to help in understanding a complex enterprise system is to seperate the system's activities, and the operations that implement those activities, into layers-called levels. This is called an abstraction hierarchy.

● Viewing a system's behavior at different level is called hierarhchical viewing.

● To build hierarchical views we must first define an abstracttion hierarchy.○ Operational description○ Hierarchical structuring

The Power of Event

Chapter 4

Designing the Electronic Enterprise


Contents

● Rapid process modification to meet the demands of the eMarketplace

● The roles of process architecture in the process lifecycle● Using process architectures to reduce errors in mission-

critical process systems● Constituents of process architecture-reactive,

behaviors, design constraints● Dynamic process architectures● Layered architectures and plug-and-play systems● The abstraction principle for layered architectures

Skip...Business related...

The Power of Event

Chapter 5

Events, Timing, and Causality


Contents

● What events are● How events are created● The form, signinficance, and relativity of an event● Timing, causality, and aggregation● Genetic parameters of events● Partial orderings of events● Timing requirements expressed as patterns of events● Examples of causal tracking of interprocess activity

5.1 What Events are

● An event is an object that is a record of an activity in a system. An event may be reated to other events.

● An events has three aspects○ Form: The form of an event is an object.○ Significance: An event signifies an acitivity.○ Relativity: An activity is related to other activities by

time, causality, and aggregation.

5.2 How Events Are Created

● Two Steps to create events1. Observation step2. Adaptation step

● Three prinicipal sources of events1. IT layer2. Instrumentation3. CEP

5.3 Time, Causality, and Aggregation

● Time: Time is a relationship that orders events● Cause: If the activity signified by event, A had to

happen in order for the activity signified by event B to happen, then A caused B. Causality is a dependence relationship between activities in a system.

● Aggregation: If event A signfies an activity that consist of the activities of a set of events, B1, B2, B3, ..., then A is an aggregation of all the events Bi. Conversely, we say events A and B are indepenent if neither caused the other.

5.3.1 The Cause-Time Axiom

● All these relationship between events have some simple mathematical properties. They are transitive and asymmetric

● Cause-time axiom: If event A caused event B in system S, then no clock in S gives B an earlier timestamp than it gives A.

5.4 Genetic Parameter in Events

● Special data parameters are added to an event to encode its timing and its casual relationships to other events. They are often referred to as genetic parameter

● Timestamps

● Casual vector: A parameter of an event that contains the identifiers of the set of events that caused the event.

5.5 Time 1/2

● A timing relationship between events created by a system tells how the system is performing.

● Timing is also an important filter in debugging● By using CEP, we can easily deal with the issue that

their order of observation might be different from their order of execution.

● The concept that makes this possible is event pattern matching.

● Event patterns are a powerful way to state our deadline requirement simply and concisely.

5.5 Time 2/2

● CEP rules like this example are called constraints because their purpose is to check a system's performance for violation of requirements

Rule: Check Timely Delibery

Element Declarations

Variable Subject S, Message M, String Id, Time T, T1, T2

Event types Publish(Subject S, String Id, Message M, Time T),Receive(Subject S, String Id, Message M, TIme T),Warning(String Id, Time T)

Relational operators and

Pattern Publish(StockTrade, Id, M, T1) and Receive(StockTrade, Id, M, T2)

Context test T2 - T1 >= 10 mins

Action create Warning(Id, T2 - T1)

5.6 Causality and Posets

● Events in a distributed system happen in a relationshp of dependence or independence. This relationship, called causality, plays an important role in any kind of analysis of what happened in a system, either online or postmortem.

● A set of events together with their causal relationship is called poset, partially ordered set of events

● The essential role of causality is to focus the search space for the causes of some activity.

5.7 Casual Event Execution-Real-Time Posets

● A causal event execution is a poset consisting of events generated by a system and their relationships.

● We use "causal event execution" tp emphasize that we are dealing with events and their relativities online real time.

5.8 Orderly Observation

● If their time of arrival is consistent with their causal relationship in the target system, we say that the observation is orderly.

● where A --> B means "A causes B", and the ArrivalTime of an event is its time of arrival at a CEP adapter, assuming adapters have synchronized clocks.

Orderly observation: for any pair of events A and B,if A --> B then ArrivalTime(A) <= ArrivalTime(B)

5.9 Observation and Uncertainty 1/2

● The events created by CEP can be thought of as event inferences drawn from observed events.

● CEP may also process these event inferences and use them to create more inferences.

● Event inferences can signify activities within the system that were not signified by events observed in the system itself.

5.9 Observation and Uncertainty 2/2

● The totality of what we can observe, together with what we can infer, is called observable system.○ Observable System: The set of all events that can

be observed from a target ssytem, or inferred from observations, is called the observable system.

○ Uncetainty Principle■ An activity within a target system may not have

any signifying event in the observable system.■ We must always bear in mind that there may be

activities going on in the system that we cannot know about.

The Power of Event

Chapter 6

Event Patterns, Rules, and Constraints


Contents

● Familiar kinds of pattern searching● Event patterns● A strawman event pattern language● Event pattern rules● Event pattern constraints● Capturing business rules as event patterns

6.1 Common Kinds of Pattern Searching

● We need to be able to describe a pattern of events that we are interested in and quickly find the sets of events that match the pattern.

● To do this, we first need a precise method to describe event patterns.

● One way is to write the pattern in a computer language called an event pattern language(EPL).

● Another way, which many of us are familiar with, is to use a graphical user interface(GUI) such as those provided in popular Web search engines.

● Unfortunately, search GUIs usually allow only very simple patterns to be described

6.2 Event Patterns 1/2

● An event pattern is a template that matches certain sets of events - the sets you want to find.

● It describes precisely not only the events but also their causal dependencies, timing, data parameters, and context.

● So an event pattern is a template for posets.

6.2 Event Patterns 2/2

● Some examples of event patterns are the following:a. All orders from custom C in the last month (content-sensitive)b. All orders from frequent customers in the last month (context-

sensitive)c. All orders from customers in response to a discount

announcement (filter)d. All orders from customers at the regular price that have led to

the customer requesting a reduced price in response to the discount announcement (complex)

6.3 A Strawman Pattern Language

● A pattern in STRAW-EPL must declare the following four elements:○ A list of variables.○ A list of types of events.○ A pattern.○ A condition on the context of any match

6.4 Event Pattern Rules 1/2

● An event pattren rule is reactive rule that specifies an action to be taken whenever an event pattern is matched.

● An event pattern rule implies a causal relationship between the events that trigger it by matching its pattern and the events that are created when the rule executes its action.

● A reactive rule has two parts:○ A trigger, which is an event pattern○ An action, which is an event that is created

whenever the trigger matches

6.4 Event Pattern Rules 2/2

● Reactive rules can be either sequential or parallel.○ A sequential rule implies that all its triggerings take

place in a sequence, one after the other.○ A parallel rule implies that its triggerings take place

independently, as if executed by new threads of control.

○ A sequential rule uses create to indicate its action, whereas a parallel rule uses create parallel.

6.5 Constraints 1/2

● Constraints can be used to specify not only how a target system should behave, but also how its user should use it.

● A never constraint consists of the following:○ The temporal operator never○ A STRAW-EPL pattern

6.5 Constraints 2/2

● The purpose of a rule is to create new events in response to situations.

● The purpose of a constraints is different, simply to monitor for a situation.

● Typically, a constraint is used to express a requirement on system behavior that is not guaranteed by the system.

The Power of Event

Chapter 7

Complex Events and Event Hierarchies


Contents

● Event aggregation● Complex events● Event abstraction hierarchies● Personalized and role-baed viewsof hierarchical

systems

7.1 Aggregation and Complex Events

● A complex event is an aggregation of other events, called its members as defined in Section 5.3.

● The relationship between a complex event and its members is called aggregation.

● A complex event can signify an acitivity that consists of several activities in different parts of a distributed system.

● Conceptually, we think of a complex event as an event at a higher level thatn the levels of its members.

7.2 Creating Complex Events 1/3

● Event pattern rules are used in CEP to create complex events siginifying the acitivities of sets of events.

● This user of event pattern rules gives us a powerful method of recognizing significant high-level events from among a cloud of low-level events.

● Aggregation is a tool for making the activities in a complex system understandable to humans.

7.2 Creating Complex Events 2/3● One of the reasons we need event patterns to create complex

events is that a complex event can happen in more than one way.● We need more powerful pattern lanuguage than STRAW-EPL.● The lower-level events that are aggregated by a complex event can

be thought of as causing the complex event.● Our primary use of complex events is to view the activities in the

system as happening at a series of abstraction levels. ● Hence we distinguish between causality, which is a relationship

between events at the same level of abstraction, and aggregation, which is an abstraction relationship.

● We sometimes refer to aggregation later on as vertical causality.

7.2 Creating Complex Events 3/3● Example

Rule: Completed Data Transfer

Element Declarations

Variable Node N1, N2, Data D, Bit B, Time T1, T2, T3, T4

Event types Send(Node N1, Node N2, Data D, Bit B, Time T1),Receive(Node N1, Node N2, Data D, Bit B, Time T1),Ack(Node N1, Node N2, Bit B, Time T1),RecAck(Node N1, Node N2, Bit B, Time T1),CompletedTrans(Node N1, Node N2, Data D, Time T1, Time T2)

Relational operators -> (causes)

Pattern Send(N1, N2, D, B, T1) -> Receive(N2, N1, D, B, T2) -> Ack(N2, N1, B, T3) -> RecAck(N1, N2, B, T4)

Context test T4 - T1 <= TimeBound

Action create CompletedTrans(N1, N2, D, T1, T4)

7.3 Event Abstraction Hierarchies● In CEP an event abstraction hiearchy consists of the following

elements.a. A sequence of levels of activities: Each level consists of a set

of descriptioins of system acitivities and, for each activities, a specification of the types of events that signify instances of that acitivity. Level 1 is the lowest leve

b. A set of event aggregation rules for each level: For each level(except level 1), there must be a rule for creating each type of event at that level as an aggregation of events at levels below.

● The crucial aspect of this definition is the set of rules specifying how each event at a higher level is an aggregation of events at levels below it.

7.4 Building Personalized Concept Abstraction Hierarchies● We use event abstraction hierarchies to specify and implement

personalized views of a target system for each stakeholder in the system.

● This is a two step process following the definition in Section 7.3.● Aggregation rules are the key to the second step.

a. The types of events that we choose to view at each hierarchical level must be specified as types of CEP events. This is the step where "personalization" of views takes places.

b. An aggregation rule must be defined for each type of event at each level above level 1.

Part 1 A Simple Introduction to Complex Event Processing

Technology

electronic enterprise

event causality

cloud of events

enterprise event traffic

single event

event aggregation

lowerlevel events

eventdriven systems