Slide 1

Jolyon WhiteGEC9, 4th November 2010Measurement Flow Architecture in OML

1OML = Measurement Flows2

Rutgers University, New Jersey

NICTA, Sydney

Deutsche Telekom Labs@ TU Berlin

BOWL TestbedNational Broadband Network100Mbs FTTHVoD TrialIREELNetwork EducationTeaching PlatformRail Bridge Monitoring SensorsNSW Road Traffic Authority

Parking DiscoveryRutgersMarco GruteserCurrent OML data pipelineApplication or ServiceMeasurement pointsFiltersMeasurement streamsOML ServerDatabase(SQL)Database tablesFileOML client library3Application writer instruments application by defining measurement points * MPs accept typed tuplesOML client library provides filtering * Application user configures the collection process * what filters, what destination

Each measurement stream can be sent to different destination

Database table schemas are created dynamically * low overhead for admin/user3SchemasSchemas enable:ProvenanceProcessing in the pipeline (data crunching)Measurement Stream schema == Combination of schemas of filter outputsEach MS stored in its own DB table4MP (A, B, C)ABC(S, T)(U, V, W)(X, Y)(S, T, U, V, W, X, Y)MS SchemaSchemasExample: app name is otr2

SQL issued to the database:

Schema names + metadata define provenance5avgavg : DOUBLEmax : DOUBLEmin : DOUBLEts : DOUBLEflow_id : INT32seq_no : UINT32pkt_length : UINT32src_host : STRINGsrc_port : STRINGMP udp_in:CREATE TABLE otr2_udp_in ([METADATA COLS], pkt_length_avg REAL, pkt_length_max REAL, pkt_length_min REAL);Measurement Collection GraphModularize producers + consumersMeasurement Point (MP) data sourceProcessing Point (PP) buffer, select, filter, join, forwardTermination Point (TP) persistent storage

6MPMPMPPPPPTPTPTPPPMetadata StoreServicesAPIMDA(Measurement Data Archive)Placing a measurement collection element at the node allows us to do processing appropriate to the type of data and it purpose. It can also help overcome limitations of the node itself.

For instance, high-resolution

We replace the simple proxy server with a fully-capable OML server, and add the ability to run periodic queries on the raw measurement tables.The result of the query is stored in another table, possibly after some custom filteringThe important thing to note is that this second table can either be on the same node, or on a remote node.

We can extend this concept by building chains of these periodic query filters to create a hierarchy of collection servers tailored to multiple different users of the measurement data.6Resource provisioningOML has no concept of resource provisioningExperimenter obtains resources for I&M identically to experimental resourcesi.e. no distinction between I&M and experiment resourcesUser has full control over how resources usedUseful defaults, but allow more if experimenter wants itCant always cleanly separate I&M from experimentMobile wireless testbeds where I&M must share compute + network with experimentE.g. ParknetAlmost all wireless traffic was measurement flows7TransportsOML currently supports two custom procotolsText versionBinary version

Standard transports are good!We like IPFIX, aiming to support it (near future)Why? Several reasons but:Template support self-describing measurement streams

8Metadata headers(schemas)Measurement flowMetadata headers(schemas)Measurement flowProcessing Point Applications9Proxy ServerBuffer measurements on commandDont transmit to remote serverSame protocol as serverTransparent to client applications

Proxy serverOML ServerApplicationCMD_BUFFERCMD_REPLAY10Hierarchical Measurement CollectionHigh-resolution measurements lose value over timeLocal storage may be limitedMeasuring at different granularitiesInspired by existing research in Streaming DatabasesNumerous VC-backed startups in financial data feed processing space

11Placing a measurement collection element at the node allows us to do processing appropriate to the type of data and it purpose. It can also help overcome limitations of the node itself.

For instance, high-resolution

We replace the simple proxy server with a fully-capable OML server, and add the ability to run periodic queries on the raw measurement tables.The result of the query is stored in another table, possibly after some custom filteringThe important thing to note is that this second table can either be on the same node, or on a remote node.

We can extend this concept by building chains of these periodic query filters to create a hierarchy of collection servers tailored to multiple different users of the measurement data.11Context-Driven Experiment SteeringDynamic experiments need measured context feedbackE.g. Geographic trip lines, link state feedback

12Once we added the periodic query and filter, we started thinking about what other applications this could be used for.

In particular, the filter can be used to sense the measurement environment.

This leads to the idea of providing an event notification mechanism to provide feedback to the experiments applications, and this allows the experiment to change its behavior according to the current context.

For instance, we might want to make a policy change if the quality of service is being degraded too muchOr we might want to only start a particular application when a mobile node has crossed some geographic landmark, such as a tripline12Context-Driven MeasurementEnvironment feedback can be used to influence the measurement process itself

13