The Plane distributed measurement infrastructure Overview, insights & hindsights Journée du Conseil Scientifique de l’Afnic , #JCSA2015, July 9 th 2015 Grant Agreement n. 318627 Dario Rossi Professor [email protected]http://www. telecom-paristech .fr/~drossi
36
Embed
The Plane distributed measurement infrastructure Overview, insights & hindsights Journée du Conseil Scientifique de l’Afnic, #JCSA2015, July 9 th 2015.
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
The Plane distributed measurement infrastructure Overview, insights & hindsights
Journée du Conseil Scientifique de l’Afnic , #JCSA2015, July 9th 2015
Motivations Measurement Architecture Ecosystem Example of use Summary
How to keep usershappy & engaged?
Researcher viewpoint
Today Internet
“The Internet is the first thing that humanity has built that humanity doesn't understand, the largest experiment in anarchy that we have ever had.”
Eric Schmidt – ex Google Exec. Chairman
Motivations Measurement Architecture Ecosystem Example of use Summary
3
…. issing orchestration
Internet measurementShed light on the Internet operational obscurity
Lot of measurement tools…
Plane: avoid to reinvent the wheel& assist in building automated pilots!
Motivations Measurement Architecture Ecosystem Example of use Summary
Plane:
4
5
Two kinds of measurements
• Passive– Observe network traffic
without interference– Similar to Aristotle’s
observational method
• Active– Perturb the network &
measure its reaction– Similar to Newton’s
experimental method
Ἀριστοτέλης Sir Isaac Newton@aristotle @newton
1388 372 2411 353
All science is either practical, poetical or theoretical (Metaphysics)
Every body continues in its state of rest, or of uniform motion in a right line, unless it is compelled to change that state by forces impressed upon it (Principia)
Motivations Measurement Architecture Ecosystem Example of use Summary
5
Passive m easurements
Deploypassive sensors
Extract meaningful
analytics
Post Process
Passive sensor
Raw (or half-cooked) data
Data collection
User traffic
Motivations Measurement Architecture Ecosystem Example of use Summary
6
Active m easurements
Raw (or half-cooked) data
Data collectionActuator
Control Controlactive sensors
A
Post ProcessActive probes
Perform active measurements
Extract meaningful
analytics
Motivations Measurement Architecture Ecosystem Example of use Summary
7
erging all together
Activesensor
Control
Passivesensor
Data
Integration with existingmonitoring frameworks
A
Active and passive analysisfor iterative root-cause-analysis
Motivations Measurement Architecture Ecosystem Example of use Summary
8
~~~Big data hype~~~
• Traffic measurements orders of m agnitude – 40Gbps (157PB/yr) full-packet processing at each passive
sensor, with up to O(107) traffic classifications per second…– O(109) active probes in an Internet anycast census…
(more on that later if time allows)
• To be compared with – The Large Hadron Collider (LHC), generates ~25PB/yr
and O(108) collisions per second– Sloan Digital Sky Survey (SDSS), generates ~73TB/yr– Capacity of the Human genome with 2-bits bases ~ O(109)
(the Big Data moment)
[Image from http://irevolution.net/category/big-data/][courtesy of Sony Classical]
Note: simplified view, yet the devil hides in the details
Plane architecture walkthrough
ARepositoriesProbes
SSupervisor
Client/Reasoner
Motivations Measurement Architecture Ecosystem Example of use Summary
10
Plane architecture: entities
ARepositoriesProbes
SSupervisor
Client/Reasoner
Orchestrate measurements; expert system (automatic pilot)
Access control; delegation; boostrap
Past sensor data; legacy databases;
Big data processing
Sensor data;Proxies for
legacy tools/platforms
Motivations Measurement Architecture Ecosystem Example of use Summary
11
Plane architecture: interfaces
AProbes
SSupervisor
Client/Reasoner
Native probe
Proxy probe/repo
Native repo
Repositories
(avoid reinventing the wheel!)
Motivations Measurement Architecture Ecosystem Example of use Summary
12
Plane architecture: messages
ARepositoryProbe
SSupervisor
Client/Reasoner
Schema-centric definition, with a type registrySchema completely describesthe measurement, JSON format
C
S
R
Motivations Measurement Architecture Ecosystem Example of use Summary
*grin* If I'd known then that [ping] would be my most famous accomplishment in life, I might have worked on it another day or two and added some more options.
Mike Muuss - US Army Research Laboratory
Motivations Measurement Architecture Ecosystem Example of use Summary
14
Plane architecture: transport
RepositoryProbe
SSupervisor
Client/Reasoner
mPlane over: • HTTPS (HTTP over TLS)• WSS (WebSockets over TLS)• SSH (Secure Shell)
Motivations Measurement Architecture Ecosystem Example of use Summary
15
Plane architecture: workflow
RepositoriesProbes
SSupervisor
Client/Reasoner
1. Send high levelspecification
2. Decompose specification
3. Send lowlevelspecification
4. Measure
5. Indirect export of Big data
6.Analyze
7.Return results
8. Aggregate results
9.Return high-levelresults
10. Decide and iterate
Motivations Measurement Architecture Ecosystem Example of use Summary
16
Plane architecture: Inter-domain
RepositoriesProbes
SupervisorS
S
S
Client/Reasoner
Motivations Measurement Architecture Ecosystem Example of use Summary
17
The broader easurement ecosystem
• “From global measurements to local management”– 10 partners, 3.8 MEUR, FP7 STREP– More focused use case: Build a framework out of probes– Knowledge sharing (e.g., joint work, Dagsthul seminars, etc.)
• IETF Large-Scale Measurement of Broadband Performance (LMAP)– Defines the components, protocols, rules, etc., but does not specifically
target adding “a brain” to the system– Common core set, Largely interoperable
• IETF IP Performance Metrics (IPPM)– Registry related, we use its vocabulary as much as possible
• IETF IP Flow Information Export (IPFIX)– Indirect export related, active contributors
• Others in scope– IETF DOCTORS; tcpm; ConEx; NETCONF; IRTF NMRG; ETSI STQ; ITU SG12
Motivations Measurement Architecture Ecosystem Example of use Summary
18
Plane consortium
Marco MelliaPOLITO
Saverio NicoliniNEC
Dina PapagiannakiTelefonica
Ernst BiersackEurecom
Brian TrammellETH
Tivadar Szemethy NetVisor
Dario RossiENST
Fabrizio InvernizziTelecom Italia
Guy LeducUniv. Liege
Pietro MichiardiEurecom
Pedro CasasFTW
Andrea FregosiFastweb
• 16 partners– 3 ISPs– 6 research
centers– 5 universities– 2 SMEs
• FP7 IP– 11 MEUR– 3 years long– ends early 2016
Motivations Measurement Architecture Ecosystem Example of use Summary
19
Plane achievements so far• Standardization &dissemination– 3 RFCs, 5 drafts– 80+ papers, including 4 Best paper awards and
ACM SIGCOMM IMC & CoNEXT, IEEE INFOCOM• Software
https://www.ict-mplane.eu/public/software and https://github.com/fp7mplane/(Ready-to-use Virtual Machines under preparation)
• Check the demos on https://www.youtube.com/channel/UCHGS6UlUKvGZTyt5DemmPaw
“Biased sampling”: Anycast geolocation as a showcase of mPlane achievements
Note: hard to summarize!
Motivations Measurement Architecture Ecosystem Example of use Summary