Decentralized Real-Time Monitoring of Network-Wide Aggregates

Decentralized Real-Time Monitoring of Network-Wide Aggregates

Rolf StadlerMads Dam, Alberto Gonzalez, Fetahi Wuhib

KTH Royal Institute of TechnologyStockholm, Sweden

www.ee.kth.se/~stadler

Large-scale Distributed Systems and Middleware (LADIS ’08)IBM TJ Watson Research Lab, NY, Sept 15-17, 2008

Outline

A self-organizing Monitoring Layer

Continuous Monitoring of Aggregates with Accuracy Objectives (A-GAP)

Performance comparison gossip vs. tree-based monitoring (GAP vs. G-GAP)

2

Today’s Management Systemsfor Today’s Network Technologies

analyze Management System

act observe

Management intelligence outsidemanaged system.

Clear separation between management system and managed

t b d i

Managed System

system, by design.

3

Today’s Management Systemsfor Today’s Network Technologies (2)

analyze

Management System

Monitoring and configuration,generally FCAPS functions, performed on a per-device basis.

Managed System

Successful for- small number of components- small rate of change.

4

A Management Layer inside the Network

analyze

exceptionsreportspolicies

5

A Monitoring System for Large-scale Dynamic Environments

1. Engineer a self-organizing monitoring layer inside the managed system.

2 Support monitoring of aggregates in real time2. Support monitoring of aggregates in real-time. across neighborhood, domain, networksum, max, average, percentile, histogram, …

3. Provide primitives for polling, continuous monitoring, detection of threshold crossings.

4. Support controlling the performance trade-offs.accuracy, overhead, execution time, robustness

6

Continuous Monitoring of Aggregates with Accuracy Objectives (A-GAP)

7

The Problem

•Find an efficient solution for continuous monitoring of aggregates i l l d i t k i tin large-scale dynamic network environments

-Aggregation functions: SUM, MAX and AVERAGE, …-Sample aggregates: total number of VoIP flows, maximum link utilization, histogram of current load across routers in a network domain

•Key Application Areas: Network Supervision,

8

y pp p ,Quality Assurance, Proactive Fault Management

Tradeoff between Estimation and Overhead

Overhead

Management solutions deployed today usually

Estimation Error

•Management solutions deployed today usually provide qualitative control of the accuracy

•Goal: Control trade-off through error objective

9

Decentralized in-Network Aggregation

Computing Aggregates•Self-stabilizing spanning tree•Incremental in networkGlobal

Management Station

•Incremental, in-network aggregation

•Push-based

Efficient Operation•Local filters conform to error

objective

AggregatingNode

Leaf Node

Global Aggregate

PartialAggregate

Localvariable

RootPhysicalNode

4

12

1

10

3

25

10

•Adapt dynamically to network statistics

3

3

5 2

2

7

75

Local Adaptive Filters

Local variable or partial aggregate

Last update value

Filter width

Filter Exceeded: 1) Triggers an update to parent2) Filter is shifted

11

Local filter on a node• Controls the management overhead by filtering updates• Drops updates with small change to partial aggregate• Periodically adapts to the dynamics of network environment

time

Problem Formalization

Find filter widths to monitor aggregatefor a given accuracy objective, with minimal overhead

Overhead: maximum processing load ωn

over all management processesAccuracy objective: { }n

nMaxω s.t. E[|Eroot|]≤ εMinimizeaverage error

12

{ }nnMaxω s.t. p(| Eroot |>γ) ≤ θMinimize

{ }nnMaxω s.t. |Eroot|≤ κMinimize

percentile error

maximum error

A-GAP: A Distributed Heuristic

•The global problem is mapped onto a local problem for each node

i i i { }πM ( ) nnEE ≤

•Attempts to minimize the maximum processing load over all nodes by minimizing the load within each node’s neighborhood

•Filter computation: decentralized and asynchronous

Minimize { }ππ

ωMax s.t. ( ) nnoutEE ε≤

13

•Each node independently runs a control cycle:every τ seconds {

request model variables from childrencompute new filters and accuracy objectives for childrencompute model variables for local node }

λn Update rate

An Stochastic Model for the Monitoring Process

•Model based on discrete-time Markov chains

Snin

Snout

En

Enout

Gn

Fn

ωn

Updates from children

Updates to parent Step sizes

Estimation Error

Update rate(processing load)

Step sizes

E ti ti E

Node state

Filter width

•It relates for each node n-the error of its partial aggregate-evolution of the partial aggregate-the rate of updates n sends -the width of the local filter

•It permits to compute for each

14

Enin Estimation Error

p pnode

-the distribution of estimation error -the protocol overhead

Stochastic Model (leaf node)

⎩⎨⎧ ≤+≤−+

=.0 otherwiseFXiFXi

jnnnnnn

n

Estimating step size (MLE)

Evolution of local variable

nX

Transition Matrix

Step Size

⎩⎨⎧

=≠≤

−−<<−+−=−=

=0

0,)()(

)(n

nnn

nnnnnnn

nnnnij j

jFjiFXiFPiXP

ijXPt

⎪⎪

⎪⎪⎪

⎨

⎧

=−==

>−==

== ∑ ∑

∑

−=

+

−=

+

−=

0)()(

)()(

)( szdGPzXP

FszsGPzXP

sSPn

n

n

n

n

n

F

Fd

Fd

Fdz

nn

Fs

Fsz

nnn

nout

15

Estimation Error

Management Overhead

⎪⎪

⎩ .0 otherwise

))0(1( =−= nout

n SPλ

nnout GE =

A-GAP: Model-based Monitoring

0,025

0,03

0,035

Error Objective

Estimation Error

0

0,005

0,01

0,015

0,02

-100 -75 -50 -25 0 25 50 75 100Estimation model variables

Optimization Problem

Stochastic Modelof Monitoring Process

Filter Widths

Overhead

0

0,5

1

1,5

2

2,5

3

node 1 node 2 node 3 node 4 node 5 node 6 node 7

Upd

ates

/sec

Measured Estimated

Step Sizes

Measurementslocal variables

Tree-based aggregation

AggregateEstimation

xi(t)

16

A-GAP: Evaluation through Simulation

•Overlay topologies -Abovenet: 654 nodes, 1332 linksGrids: 25 85 221 613 nodes-Grids: 25, 85, 221, 613 nodes

•Aggregate: Number of http flows in the domain•Traces

-From two 1 Gbit/s links that connect University of Twente to a research network

•Control cycle

17

y-τ=1 sec

Tradeoff: Accuracy vs Overhead

400

500

600

ec

ARCε =0

0

100

200

300

0 5 10 15 20 25 30

Upd

ates

/se

Tmin=0.03

0.100.05

0.04

0.20

A-GAP

ε =2

ε =5

ε =10 ε =15 ε =20

18

• Overhead decreases monotonically

• Overhead depends on the changes of the aggregate, not on its value.• A-GAP outperforms a rate-control scheme (ARC)

0 5 10 15 20 25 30Avg Error

Scalability

0,8

1

aliz

ed)

Grid 25Grid 221

Grid 613

0,2

0,4

0,6

Upd

ates

/sec

(nor

m

emin

19

• Minimum error emin increases with the network size • Overhead increases linearly with network size for same error objective

00 5 10 15 20 25 30 35 40

Avg Error

Robustness

500

1500

2500

3500

Estim

atio

n Er

ror

40

60

80

m L

oad

(Upd

ates

/sec

)

-500140 145 150 155 160 165 170 175Time

Node A fails End of Transient

20

• Estimation error: several spikes during sub-second transient period • Overhead: single peak with a long transient

0

20

140 145 150 155 160 165 170 175Time

Max

imum

Error Prediction by A-GAP vs Actual Error

0,04

0,05 Absolute Avg Error

A t l E

Error Predictedby A-GAP

0,01

0,02

0,03

Actual Errory

21

• Accurate prediction of the error distribution• Maximum error >> average error (one order of magnitude)

0-40 -30 -20 -10 0 10 20 30 40Error

Management Station

A-GAP Prototype

Lab testbed at KTH•16 monitoring nodes

Aggr

egat

ion

Tree

Node 1

Node 2 Node 3

Node 4 Node 5 Node 6 Node 7

•16 Cisco 2600 Series routers•Smartbits 6000 traffic

generator•A-GAP implemented in Java

22

Phy

sica

lN

etw

ork

Prototype: Management Station Interface

SelectAggregation Function

Evolution of the Aggregate

(True Value and A-GAP Estimation)

SelectAccuracyObjective

Show Aggregation Tree

A GAP Estimation)

Overhead Distribution and Evolution

SelectRoot Node

23

Tree

Real-time Estimation ofError Distribution and Trade-off

Simulation vs Testbed Measurements

10

12

14

TestbedExperiment

0

2

4

6

8

Upd

ates

/sec

Simulation

24

•Curves are close: difference in overhead below 3,5%•Prototype validates simulation mode

0 2 4 6 8 10 12Avg Error e

Prototype: Error Estimation by A-GAP vs Actual Error

0,08

0,10

Measured Error Error Estimatedby A-GAP

0,02

0,04

0,06

Absolute Avg Error

25

•Accurate estimation of the error distribution•Maximum error >> average error (one order of magnitude)

0,00-30 -20 -10 0 10 20 30Error

g

Prototype: Overhead Estimation by A-GAP vs Actual Overhead

2,5

3

0

0,5

1

1,5

2

Upd

ates

/sec

26

•Accurate estimation of the overhead•Estimation tends to be more accurate for

nodes close to the root

node 1 node 2 node 3 node 4 node 5 node 6 node 7

Measured Estimated

Gossip vs. Tree-based Aggregation

27

Gossip protocols

Gossip protocols are round-based,during each round a node randomly selects aduring each round a node randomly selects a subset of neighbors and interacts with them.

Applications- information dissemination- database replication - failure detection- failure detection- resource discovery- computing aggregates- …

28

Computing aggregates with gossiping

Push Synopses [Kempe et al. ‘03]

• The protocol

Round 0 { 1.

ii xs = ;

2. 1=iw ;

computes AVERAGE of the local variables xi.

• After each round a new estimate of the aggregate is computed as si/wi.

• Exponential convergencefor uniform gossip and

3. send ),( ii ws to self }Round 1+r {

1. Let * *{( , )}l ls w be all pairs sent to i

during round r 2. *

li ls s=∑ ; *

li lw w=∑

3. choose shares 0, ≥jiα for all nodes j

such that ∑ =j ji 1,α

for uniform gossip and complete graphs

• Protocol Invariants:

4. for all j send )*,*( ,, ijiiji ws αα to each j }

, ,r i r ii is x=∑ ∑ , ,r i ri

w n=∑

29

5. for all j Neighbors∈ { a. , , , ,( , ) ( , )i j i j i j i jrs rw rs rw= +

: ( )

(( ( ), ( ) ( ), ( )))m orig m j

rs m rw m acks m ackw m=

−∑

b. , , , ,( , ) ( , )i j i j i j i jacks ackw srs srw= +

( ( ) ( ))∑

Round 0 { 1. ii xs = ;

2 1

The G-GAP protocol

: ( )

( ( ), ( ))m orig m j

s m w m=∑

c. if (detected_failure(j)) { i. , ,( , ) ( , ) ( , )i i i i i j i js w s w rs rw= +

ii. , , , ,( , ) ( , ) (0,0)i j i j i j i jrs rw srs srw= =

iii. \i iL L j= } }

6. for all ij L∈ {

a. choose 0, ≥jiα such that 1, =∑ j jiα

b choose 0≥β such that

2. 1=iw ;

3. { }iL i= ;

4. for each node j )0,0(),( ,, =jiji rwrs ;

5. for each node j )0,0(),( ,, =jiji srwsrs ;

6. send )0,0,0,0,,( ii ws to self; 7. for all ij ≠ send )0,0,0,0,0,0( to j }

Round r+1 { 1. Let M be all messages received

by i during round r 2. , 1,( )( )i r i r im M

x xs s m −∈−= +∑ ; ( )i m M

w w m∈

=∑ b. choose 0, ≥jiβ such that

∑ =j ji 1,β and 0, =iiβ

c. , , , , , ,( , ) ( ), ( 1)i j i j i j i i i i i j i i isrs srw s x wβ α β α= − −

d. send , , , , , ,( , , , , , )i j i i j i i j i j i j i js w srs srw acks ackwα α to j

e. ),(),( ,,,,,, ijijiijijijiji wrwsrsrwrs αα ++=

} }

3. for all j , ,( , ) (0,0)i j i jacks ackw =

4. ( )i iL L orig M= ∪

30

Accuracy vs. Overheadgossip- and tree-based aggregation protocol

GAP and G-GAP654 node networkGoCast overlay, connectivity 10 aggregation: AVERAGEUT trace4 rounds/secno failures

31

Accuracy vs. Failure Rategossip- and tree-based aggregation protocol

GAP and G-GAP654 node networkGoCast overlay, connectivity 10 aggregation: AVERAGEUT trace4 rounds/secnodes fail randomly, y,recover after 10 sec

32

Summary

•A self-organizing monitoring layer inside the managed system

-Monitoring network-wide aggregates.Monitoring network wide aggregates. -Polling, continuous monitoring, threshold detection. -Controlling the performance trade-offs.

•Continuous monitoring of aggregates with accuracy objectives-Efficient, scalable and adaptable monitoring using aggregation trees is feasible.Model based monitoring allows for performance prediction-Model-based monitoring allows for performance prediction.

•Tree-based vs. gossip-based continuous monitoring -In a traditional wireline networking scenario, tree-based aggregation outperforms gossip-based aggregation

33

ReferencesF. Wuhib, M. Dam, R. Stadler: “Decentralized Detection of Global

Threshold Crossings Using Aggregation Trees,” Computer Networks, Vol. 52, No. 9, pp 1745-1761, 2008.

A. Gonzalez Prieto, R. Stadler: “A-GAP: An Adaptive Protocol for , pContinuous Network Monitoring with Accuracy Objectives,” IEEE Transactions on Network and Service Management (TNSM), Vol. 4, No. 1, June 2007.

F. Wuhib, M. Dam, R. Stadler, A. Clemm: “Robust Monitoring of Network-wide Aggregates through Gossiping,” 10th IFIP/IEEE International Symposium on Integrated Management (IM 2007), Munich, Germany, May 21-25, 2007.

K.S. Lim and R. Stadler: “Real-time views of network traffic using decentralized management,” 9th IFIP/IEEE International Symposium on Integrated Network Management (IM 2005), Nice, France, May 16-19, 2005.

34