Measurement Flow Architecture in OML

Jolyon WhiteGEC9, 4th November 2010

Measurement Flow Architecture in OML

OML = Measurement Flows

2

Rutgers University, New Jersey

NICTA, Sydney

Deutsche Telekom Labs@ TU Berlin

BOWL Testbed

National Broadband Network100Mbs FTTH

VoD Trial

IREELNetwork EducationTeaching Platform

Rail Bridge Monitoring Sensors

NSW Road Traffic Authority

Parking DiscoveryRutgers

Marco Gruteser

Current OML data pipeline

Application or

Service

Measurement points Filters Measurement streams

OML Server

Database

(SQL)

Database tables

File

OML client library3

Schemas

• Schemas enable:– Provenance– Processing in the pipeline (data crunching)

• Measurement Stream schema == Combination of schemas of filter outputs

• Each MS stored in its own DB table

4

MP (A, B, C) A

B

C

(S, T)

(U, V, W)

(X, Y)

(S, T, U, V, W, X, Y)

MS Schema

Schemas

• Example: app name is “otr2”

• SQL issued to the database:

• Schema names + metadata define provenance5

avg avg : DOUBLEmax : DOUBLEmin : DOUBLE

ts : DOUBLEflow_id : INT32seq_no : UINT32pkt_length : UINT32src_host : STRINGsrc_port : STRING

MP udp_in:

CREATE TABLE otr2_udp_in ([METADATA COLS], pkt_length_avg REAL, pkt_length_max REAL, pkt_length_min REAL);

Measurement Collection Graph

• Modularize producers + consumers• Measurement Point (MP) – data source• Processing Point (PP) – buffer, select, filter, join,

forward• Termination Point (TP) – persistent storage

6

MP

MP

MP

PP

PP

TP

TP

TP

PP Metadata Store

ServicesAPI

MDA(Measurement Data Archive)

Resource provisioning

• OML has no concept of resource provisioning• Experimenter obtains resources for I&M identically

to experimental resources– i.e. no distinction between I&M and experiment resources

• User has full control over how resources used• Useful defaults, but allow more if experimenter

wants it• Can’t always cleanly separate I&M from

experiment– Mobile wireless testbeds where I&M must share compute

+ network with experiment– E.g. Parknet

• Almost all wireless traffic was measurement flows7

Transports

• OML currently supports two custom procotols– Text version– Binary version

• Standard transports are good!• We like IPFIX, aiming to support it (near future)• Why? Several reasons but:

– Template support self-describing measurement streams

8

Metadata headers(schemas) Measurement flow

Metadata headers(schemas) Measurement flow

Processing Point Applications

9

Proxy Server

• Buffer measurements on command– Don’t transmit to remote server

• Same protocol as server– Transparent to client applications

Proxy server OML ServerApplication

CMD_BUFFERCMD_REPLAY

10

Hierarchical Measurement Collection

• High-resolution measurements lose value over time

• Local storage may be limited• Measuring at different granularities• Inspired by existing research in Streaming

Databases– Numerous VC-backed startups in financial data feed

processing space

11

Context-Driven Experiment Steering

• Dynamic experiments need measured context feedback

• E.g. Geographic trip lines, link state feedback

12

Context-Driven Measurement

• Environment feedback can be used to influence the measurement process itself

13