The Query Mesh Project: A Powerful Multi-Route Query Processing Paradigm New England Database Summit 2010 Elke. A. Rundensteiner Worcester Polytechnic Institute [email protected]Elisa Bertino Purdue University [email protected]1 Rimma V. Nehme Microsoft Jim Gray Systems Lab [email protected]Thanx goes to NSF 0917017 for partial support of this project.
21
Embed
1 Elke. A. Rundensteiner Worcester Polytechnic Institute [email protected] Elisa Bertino Purdue University [email protected] 1 Rimma V. Nehme Microsoft.
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
1
The Query Mesh Project: A Powerful Multi-Route Query Processing
Paradigm
New England Database Summit 2010
Elke. A. RundensteinerWorcester Polytechnic Institute
Single Lightweight Operation to Physically Adapt QM
.
.Self-Tuning Query Mesh
…
…
…
Contributions
13
[NRB09] R. Nehme, E. Rundensteiner and E. Bertino, Self-Tuning Query Mesh for Adaptive Multi-Route Query Processing, In EDBT 2009.
14
ST-QM Architecture
Static QM Framework
Query Executor
Query Optimiz
er
Query Executor
Query Optimiz
er
ST-QM
Adaptive QM Framework
14
[NRB09] R. Nehme, E. Rundensteiner and E. Bertino, Self-Tuning Query Mesh for Adaptive Multi-Route Query Processing, In EDBT 2009.
15
ST-QM Monitor continuously samples data and execution statistics that will be used to determine if a concept drift has occurred (i.e., QM needs to be adapted)
ST-QM Analyzer determines if a concept drift has actually occurred and makes recommendations if and how the QM solution should be adapted
ST-QM Actuator takes these recommendations and physically adapts the QM solution
ST-QM Components
ST-QMMonitor
ST-QMAnalyzer
ST-QMActuator
measurements recommendations
actuationsampling
15
Query Mesh
ST-QM
NewQuery Mesh
17
Classifier Modification
Query Mesh
………
Query Mesh
………
Query Mesh
………
R1 New Classifier + Old Routes
R2 Old Classifier + New Routes
R3 New Classifier + New Routes
ST-QM Actuator: Physical Query Mesh Adaptation
All possible recommendations:Case 1: Virtual Concept Drift RecommendationCase 2: Real Concept Drift RecommendationCase 3: Hybrid Concept Drift Recommendation
1
2
3
4
0
…
Query results
OI-arrayOp-modules
opi
opi
opk
opl
Self-Routing Fabric
Data
r1
r2
r3
r1
r2
r3
Online Classifier
rusters
rusters
CurrentClassifier
NewClassifier
The beauty of
the proposed design!!!
17
18
Experimental Evaluation
ST-QM was implemented inside Java-based continuous query engine called CAPE
Compare its relative performance against competitor systems, namely, we compared adaptive QM against: Static (non-adaptive) QM, Adaptive “plan-less” Eddies Adaptive “plan-less” Eddies with CBR-based routing policy
Results can be found in EDBT’ 2010.
18
19
Summary of ST-QM Experimental Results
ST-QM gave up to 44% improvement in execution time and output rate compared to non-adaptive QM, Eddy and single plan execution approach
The runtime overhead of ST-QM relative to query execution is small (on average 2%).
The actuation cost of physical adaptivity is nearly negligible resulting in 0.02% of total execution cost
Even if no adaptivity is needed, ST-QM’s performance in the worst case will be at most 2-3% slower than static QM
19
20
Conclusion
• Query Mesh is practical query optimization approach Eliminates single plan assumption Feasibility shown Has low overhead & high potential benefit Easily implemented and integrated with existing
systems
• Query Mesh leads to novel solutions Usage of machine learning in query optimization
and query processing Usage of network-inspired techniques in query
optimization and query processing20
21
Next Steps in QM Project
• Consider state caching and indexing in QM stream context
• Work with alternate classification methods for route decisions
• Design customized query optimization and processing strategies
• Study multi-query processing and optimization
• Scale by applying distributed processing technologies
• Do QM principles also apply in static DB context !?
21
22
Thank You for Listening !!!!!
22
Thank you to current and past DSRG members for stream engine development, feedback, collaboration, and much more.