Communication- Efficient Distributed Monitoring of Thresholded Counts Ram Keralapura, UC-Davis Graham Cormode, Bell Labs Jai Ramamirtham, Bell Labs
Communication-Efficient Distributed Monitoring of Thresholded Counts Ram Keralapura, UC-Davis Graham Cormode, Bell Labs Jai Ramamirtham, Bell Labs.
Dec 20, 2015
Welcome message from author
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
Communication-Efficient Distributed Monitoring of Thresholded Counts
Ram Keralapura, UC-Davis
Graham Cormode, Bell Labs
Jai Ramamirtham, Bell Labs
June 28, 2006 Ram Keralapura, UCDavis 2
Introduction
Monitoring is critical to managing distributed networked systems
Main challenges: Continuous Distributed Resource-constrained environments
June 28, 2006 Ram Keralapura, UCDavis 3
Thresholded Counts
New fundamental class of problems “Tracking counts for an event beyond a given
threshold value with user-specified accuracy” Motivating scenarios:
Total # of connections to a server when it exceeds the normal operational condition (ex, DDoS attacks)
Total traffic to a particular destination prefix when it exceeds the pre-defined limit
Tracking the total number of cars on a highway
June 28, 2006 Ram Keralapura, UCDavis 4
Thresholded Counts (cont’d)
Two key properties Threshold value User specified tracking accuracy
TNNNN
TNTN
when ˆ)1(
when ˆ0
June 28, 2006 Ram Keralapura, UCDavis 5
System Architecture
Remote Site 1
Remote Site m
Remote Site 2
Remote Site i
Coordinator Site(Central Node)
ivc ,
mvc ,
2,vc
1,vc
remote sites (or monitors) and a coordinator site (or central node)m
Non-continuous updates
Local thresholds at remote sites
Counts can be positive, negative, or fractional
Ignore network delays and losses
June 28, 2006 Ram Keralapura, UCDavis 6
m
iii,ftN
1)(
ˆ
Every remote monitor , maintains a set of local thresholds:
Local count at monitor , should always lie between two neighboring thresholds
Global estimate at the central node:
...,,jt ji 210 ,,
Approach
1)()( ii,fiii,f tNt
i
i
June 28, 2006 Ram Keralapura, UCDavis 7
Approach (cont’d)
Maximum error in the global estimate should satisfy:
Two methods to set local thresholds Static thresholding Adaptive thresholding
TNNttm
iii,fii,f
when )(0
1)(1)(
June 28, 2006 Ram Keralapura, UCDavis 8
Static Thresholding
Problem: For given values and , we have to determine such that,
T ),0[ ,, jt ji
0 and :0 0,,1, ijiji tttj
Tttttm
iii,f
m
iii,f
m
iii,fii,f
11)(
1)(
1)(1)( when )(
N
N̂
Uniform
Proportional
TN
N̂
T
1N2N
1N
3N
2N3N
N
N̂3N
3N
N
N̂
m
T
m
T2
)1(
0
01
2)1(
Monitor-1
Monitor-1
Monitor-2
Monitor-2 Monitor-3
Monitor-3
Central Node
Central NodeBlended threshold assignment
TMax error =
Max error = N
June 28, 2006 Ram Keralapura, UCDavis 10
Static Thresholding (cont’d)
Blended threshold assignment
uniform threshold assignment proportional threshold assignment
Complexity:
1 ,1 when and 0
10 where)1()1(
1,0,
1,,
ii
jiji
ttm
Ttt
01
11logT
NmO
June 28, 2006 Ram Keralapura, UCDavis 11
Adaptive Thresholding
Every monitor maintains only two threshold values: and
Problem: For given values and , and a threshold violation from monitor , determine for all the monitors such that,
T iLt iHt
k iHt
iLiH tti :
when 111
Tttttm
iiH
m
iiL
m
iiLiH
Lt3
Lt2
Slack
Lt1
Monitor-1 Monitor-2 Monitor-3 Central Node
T
N̂Ht3
Ht2Ht1
Lt3
Lt2Lt1
Monitor-1 Monitor-2 Monitor-3 Central Node
T
N̂Ht3
Ht2Ht1
TN )1(ˆ
TN )1(ˆ
Basic Adaptive Algorithm
June 28, 2006 Ram Keralapura, UCDavis 13
Experimental Setup
Built a simulator with monitoring nodes and a central node
Implemented all the static and adaptive algorithms
Data set: Public traces from NLANR
m
June 28, 2006 Ram Keralapura, UCDavis 14
Count Accuracy
June 28, 2006 Ram Keralapura, UCDavis 15
Validating the Theoretical Model
],,[ NT
],,[ NT
June 28, 2006 Ram Keralapura, UCDavis 16
Comparing Costs – Static and Adaptive Cases
],,[ NT
June 28, 2006 Ram Keralapura, UCDavis 17
Related Work
Top-k monitoring [Babcock et al] Heavy-hitter definition Adaptive filters for continuous queries [Olston et al]
Distributed continuous queries but does not address the thresholded counts problem
Distributed triggers [Jain et al] Simplified version of the thresholded counts problem Randomized algorithms with statistical guarantees
Geometric approach for threshold functions [Sharfman et al] Focus is mainly on non-linear functions
June 28, 2006 Ram Keralapura, UCDavis 18
Summary
We defined a fundamental class of problems called “Thresholded Counts”
We proposed algorithms to address the problem – static and adaptive
Analyzed the complexities of these algorithms and provided proofs
Using experiments, we showed the effectiveness of our algorithms
June 28, 2006 Ram Keralapura, UCDavis 19
Future Work
Building the monitoring system for real networks to explore the practical aspects of our framework Sensor networks IP network monitoring
Address scalability issues For example, hierarchical monitoring architecture
Extend for different query types with thresholded nature For example, arithmetic combinations
Related Documents
