Efficient Distribution-Based Event Filtering Annika Hinze, Sven Bittner Institute of Computer Science Freie Universität Berlin {hinze, bittner}@inf.fu-berlin.de.

Efficient Distribution-Based Event Filtering Annika Hinze, Sven Bittner Institute of Computer Science Freie Universitt Berlin {hinze, bittner}@inf.fu-berlin.de Talk at the workshop on Distributed Event-based Systems At the Conference ICDCS 02.07.2002

29.06.2002Efficient Distribution-Based Event Filtering 2/18 Event Notification Service Client Repository Profile Repository Event Message Repository Notification Provider Invoker Repository Information Object ObserverFilterNotifier Introduction: Components of an ENS

29.06.2002Efficient Distribution-Based Event Filtering 3/18 Introduction: Filtering Algorithms - Related work Performance analysis of filter algorithms Algorithms: - Tree-based algorithm by Gough et al [gough95] - Tree-based algorithm by Aguilera et al [aguilera99] - Counting algorithm [yan94] - Equality-preferred algorithm [fabret00] - Inequality-preferred algorithm [fabret00] -Sequentially-test-all algorithm Evaluations based on equally distributed data

29.06.2002Efficient Distribution-Based Event Filtering 4/18 Introduction: Filtering Algorithms Problem statement Applications: Environmental Monitoring: Events: Sensor readings produce wide range of data Profiles: Catastrophe Warning (not likely, but important) Logistics Support: Events: Car Locations, accident information Profiles: diverse, filtering+routing Extreme distributions for events and profiles -High peaks for small range -Gauss distributed data,

29.06.2002Efficient Distribution-Based Event Filtering 5/18 Introduction: Filtering Algorithms Problem statement Filter algorithms: performance differs with distributions! Goal: Improving filter performance for certain applications based on the typical distributions

29.06.2002Efficient Distribution-Based Event Filtering 6/18 Roadmap Motivation Recap: Tree-based Algorithm Performance Model Measures for filter algorithms Test results Conclusions from testing Outlook

29.06.2002Efficient Distribution-Based Event Filtering 7/18 20 30 35 80 90 80 85 95 P4 P3 P1 P2 P5 20 90 Recap: Tree-based Algorithm Fastest Filter Algorithm [gough95] : Profiles: P1[temp=30; hum=80] P2[temp=35; hum=85] P3[temp=20; hum=80] P4[temp=20; hum=90] P5[temp=35; hum=95] For each attribute sequentially test all branches Event: E[temp=20; hum=90] P4

29.06.2002Efficient Distribution-Based Event Filtering 8/18 Model Performance measured in comparison steps Events and profiles: certain probability P e, P p Profile values naturally ordered in tree For simplicity: Profiles possible attribute values (D 0 = ) #steps = expectation for X = E(X) = x i P e (x i ) xixi Event distribution of one attribute a: -Event modeled as value of discrete random variable X -Values of X: x i :ordering number of attribute values

29.06.2002Efficient Distribution-Based Event Filtering 9/18 Model Distributions of n attributes not independent Conditional distributions: e.g., temperature and humidity related # steps for each attribute a j : E (X j |X j-1,..,X 1 ) ( E (X j |X j-1,..,X 1 ) ) = x i j P e (x i j | x j-1,..,x 1 ) jj #steps for all attributes: Strong dependence on order -Within each tree level -Of attributes (conditional distributions !)

29.06.2002Efficient Distribution-Based Event Filtering 10/18 Measures for filter algorithm Influence algorithm by V - Reorder edges according to value selectivity 1.Event distribution 2.Profile distribution 3.Profile distribution * event distribution A - Reorder nodes according to attribute selectivity 1.# values without profiles / # all values 2.Probability of values without profiles [simplified] 3.# values without profiles - conditional distributions

29.06.2002Efficient Distribution-Based Event Filtering 11/18 Roadmap Motivation Recap: Tree-based Algorithm Performance Model Measures for filter algorithm Test results Conclusions from testing Outlook

29.06.2002Efficient Distribution-Based Event Filtering 12/18 Test setting Algorithm implementation in Java Simulated reordering All permutations of 60 distributions 8 different orderings plus binary search 4 Test Groups: -Tree of 5,000 /10,000 profiles, tests until 95% precision -Full profile tree (n attributes), tests until 95% precision -Full profile tree, one attribute only, 4000 events -Full profile tree, one attribute only, all possible events

29.06.2002Efficient Distribution-Based Event Filtering 13/18 Test results: Value reordering

29.06.2002Efficient Distribution-Based Event Filtering 14/18 Conclusions from testing: Value reordering V1 Event distribution -Faster than binary search if E (X ) < log 2 (#profiles) holds for event-distribution V2 Profile distribution and V3 Profile distribution * Event distribution -Faster notifications for profiles with high priority -Inferior average response time according to events -No queue if filter in optimal working point (f events f filter )

29.06.2002Efficient Distribution-Based Event Filtering 15/18 Test results: Attribute reordering

29.06.2002Efficient Distribution-Based Event Filtering 16/18 Conclusions from testing: Attribute reordering Idea: Reject unmatched events early A1 # values without profiles / # all values -Faster for equally distributed data A2 Probability of values without profiles -Faster for all event distributions A3 conditional distributions -Faster for all event distributions -Costly to obtain, but best measure -Only sensible for applications with stable distributions

29.06.2002Efficient Distribution-Based Event Filtering 17/18 Conclusion Fast distribution-dependent algorithm Introduction of value-dependent and attribute- dependent selectivity measures Tests show usefulness of measures Introduction of criteria for reordering circumstances under which reordering improves performance

29.06.2002Efficient Distribution-Based Event Filtering 18/18 Outlook Distribution-dependent Adaptive Service based on -Predefined distribution for events in application -History of events (to determine the event distribution) Currently: implementation of GENAS System adapts to -Distributions -Event semantics -Temporal correctnes

Thank you for your attention! Contact: Annika Hinze [email protected] Sven Bittner [email protected]

29.06.2002Efficient Distribution-Based Event Filtering 20/18 Bibliography: My Publications [hinze02] A.Hinze, A.Voisard: Composite Events in Notification Systems with Application to Logistics Support, submitted for review to Caise [faensen01] D. Faensen, L.Faulstich, H.Schweppe, A. Hinze, and A. Steidinger: Hermes -- A Notification Service for Digital Libraries, In Proceedings of the ACM/IEEE Joint Conference on Digital Libaries (JCDL), 2001, Roanoke, VA, USA, June 2001. [hinze01] A.Hinze: Does Alerting have special Requirements for Query Languages? in Martin Endig, Thomas Herstel (Hrsg.): Tagungsband zum13. GI-Workshop Grundlagen von Datenbanken.Uni. Magdeburg, Preprint Nr. 10, Juni 2001 [hinze01a] A. Hinze: How does the observation strategy influence the correctness of alerting services? Proceedings of the BTW 2001, Oldenburg, March 2001. [schweppe00] H.Schweppe, A. Hinze, D.Faensen: Event-based Notification on the Web, Tutorial at the 9th WWW9, May 15-19,2000, Amsterdam [schweppe00a] H.Schweppe, A. Hinze, D.Faensen: Database Systems as Middleware Events, Notification, Messages. ADBIS-DASFAA 2000: 21-22 [hinze99] A. Hinze, D.Faensen: A Unified Model of Internet Scale Alerting Services, Proceedings of the ICSC, December 13-15, 1999, Hong Kong, Springer-Verlag. [hinze99a] A. Hinze and H.Schweppe: Notification Services in Digital Libraries, Proc. of the Dagstuhl Seminar 99351 "Multimedia Database Support for Digital Libraries", August 1999, Dagstuhl [hinze99b] A. Hinze, D.Faensen: A Unified Model of Internet Scale Alerting Services, Technical Report, Number tr-b-99-15, Freie Universitt Berlin, 1999 [hinze99c] A. Hinze: Alerting Services in a Digital Library Environment, Doctoral Consortium at CAiSE, 1999, Heidelberg, published as Technical Report, ETH Zrich [hinze99d] A. Hinze: A Model of Alerting Services in Wide Area Networks, in F.Hsemann, K.Kspert, F.Murer (Eds.): 11. Workshop "Grundlagenvon Datenbanken", Luisenthal, 25.-28. Mai 1999. [faensen98] D.Faensen, A.Hinze, H.Schweppe: Alerting in a Digital Library environment - Do Channels meet the requirements? In Second ECDL, number 1513 inLNCS, 1998. Springer-Verlag. [faensen98a] D.Faensen, A.Hinze, H.Schweppe: Alerting in a Digital Library environment - Do Channels meet the requirements? TR, Number tr-b-98-08, Inst. of Computer Science, Freie Universitt Berlin, 1998

29.06.2002Efficient Distribution-Based Event Filtering 21/18 Bibliography: Filtering Algorithms [hanson90] E.Hanson, M.Chaabouni, C.Kim, and Y.Wang: A predicate matching algorithm for database rule systems, SIGMOD, 1990 [gough95] K.Gough and G.Smith: Efficient Recognition of events in distriuted systems, Proceedings of the ACSC-18, 1995 [aguilera99] M.K. Aguilera, R.Strom, D.Sturman, M.Astley, and T.Chandra: Matching events in a content-based subscription system, 18 th ACM PODC, 1999 [fabret00] F. Fabret, F. Llirbat, J. Pereira, and D. Shasha: Efficient matching for content-based publish/subscribe systems", technical Report, INRIA, 2000

Efficient Distribution-Based Event Filtering Annika Hinze, Sven Bittner Institute of Computer Science Freie Universität Berlin {hinze, bittner}@inf.fu-berlin.de.

Documents

e x j x j

xixi event distribution

j p e x

branches event

p4 slide

typical distributions

ens slide

filter algorithm gough95