20100211-APAN29-pSArchitecture-1.ppt

perfSONAR ArchitectureperfSONAR Architecture

February 11th 2010, APAN 29 – perfSONAR WorkshopJeff Boote, Senior Network Software Engineer

• Motivation and Goals• Service Oriented Architecture (SOA)• Architecture Overview

– Service Components– Client Examples

• Communication Protocols• Framework Interaction

– Deploying a Service– Simple Client/Service Interaction– Lookup Service Interaction

2 – 04/13/23, © 2009 Internet2

Outline

• Thus far we have seen that the perfSONAR framework can be used to solve end-to-end performance problems for multi-domain paths.

• The framework is made up of several unique components and design considerations, all of which operate in a cooperative yet independent manner– Each functionality is separated into a specific function– Clients and servers interact through scripted, XML Based

protocols– Measurement data is encoded in expressive XML formats

• By the end of this talk we will see:– A complete description of services for the entire framework– Overview of protocols– Use cases for clients and services alike

3 – 04/13/23, © 2009 Internet2

Motivation and Goals

• Interoperable network measurement middleware (SOA):– Modular– Web services-based– Decentralized– Locally controlled

• Integrates:• Network measurement tools and data archives• Data manipulation• Information Services

• Discovery• Topology• Authentication and authorization

• Based on:• Open Grid Forum Network Measurement (NM) Working Group schema• Currently attempting to formalize specification of perfSONAR protocols in

the Network Measurement Control (NMC) working group.• Network topology description being defined in the Network Markup

Language (NML) Working Group

Service Oriented Architecture (SOA)

4 – 04/13/23, © 2009 Internet2

• Measurement Point (MP) Service– Enables the initiation of performance tests

• Measurement Archive (MA) Service– Stores and publishes performance monitoring results

• Transformation Service– Transform the data (aggregation, concatenation, correlation,

translation, etc)• Resource protector

– Arbitrate the consumption of limited resources– Other services delegate a limited portion of the authorization

decision here

These services are specifically concerned with the job of network performance measurement and analysis

Service Oriented Architecture (SOA)

5 – 04/13/23, © 2009 Internet2

6

• Lookup Service– Allows the client to discover the existing services and other LS services.– Dynamic: services registration themselves to the LS and mention their

capabilities, they can also leave or be removed if a service goes down.

• Topology Service– Make the network topology information available to the framework.– Find the closest MP, provide topology information for visualisation tools

• Authentication Service– Based on Existing efforts: Internet2 MAT, GN2-JRA5– Authentication & Authorization functionality for the framework– Users can have several roles, the authorization is done based on the user

role.– Trust relationship between networks

These services are the infrastructure concerned with discovering federating the available network services

Service Oriented Architecture (SOA)

6 – 04/13/23, © 2009 Internet2

Where is link utilization for - IPs d,e,f?

Client

Network A Network B

LS A LS BMA A MA B

a bc d

e f

Where is link utilization for – IPs a,b,c?a,b,c : Network A, MA A Get link utilization d,e,fHere you goGet Link utilization a,b,c

Here you go

Useful graphgLSWhere can I get more aboutnetwork Doman B/IP d,e,f andDomain A/IP a,b,c?

LS A, LS B

d,e,f : Network B, MA B

Service Oriented Architecture (SOA)

7 – 04/13/23, © 2009 Internet2

8 – 04/13/23, © 2009 Internet2

Architecture Overview

Analysis & Visualization

Measurement Infrastructure

Data Collection Performance

Tools

Analysis & Visualization

Measurement Infrastructure

API

API

9 – 04/13/23, © 2009 Internet2

perfSONAR Architecture Overview

• Measurement Point (MP) form the lowest layer of the monitoring infrastructure– Directly interacts with the measurement tool– Can offer WS control over on-demand measurement– Can offer interface to a regular scheduled measurements

• Rolls of the Measurement Point:– Utilize well known tools to perform measurements– Offer, at a minimum, cache storage of recently performed

measurements– Interact with Measurement Archives (MAs) to archive stored

measurements• Examples:

– perfSONAR-BUOY (OWAMP and BWCTL Testing)– PingER (Ping Testing)– Command Line MP

10 – 04/13/23, © 2009 Internet2

Architecture Overview - MP

• Measurement Archive (MA) stores the results of network and performance measurements– WS interface for storage and query– Interacts with backend databases (e.g. SQL, RRD)

• Rolls of the Measurement Archive:– Expose historical and current measurements of diverse types– Draw data queries away from the Measurement Points (MPs)

• Examples:– perfSONAR-BUOY (OWAMP and BWCTL Data)– PingER (Ping Data)– SNMP MA– Status MA

11 – 04/13/23, © 2009 Internet2

Architecture Overview - MA

• The Transformation Service (TrS) performs operations on data sets (e.g. aggregation, correlation). – WS interface– Potential to store well known operations, and replay later

• Rolls of the Transformation Service :– Draw complex statistical queries from Measurement Archives– Provide a conduit for popular operations (e.g. running statistics

over several changing dataset). • Examples (Planned):

– Path diagnostics tools• Combining multiple metrics (network path, utilization, latency,

bandwidth)

– Data presentation• Statistical results for raw measurements.

12 – 04/13/23, © 2009 Internet2

Architecture Overview - TrS

• The Resource Protector (RP) monitors the relative performance and availability of the monitoring infrastructure – Knowledge of the services in a given deployment– Defined policy regarding access and resources

• Rolls of the Transformation Service :– Protects the time and resources of services from being overrun

• Too many queries from a single source• Too much data for a given query

– Cooperate with the Authentication and Authorization (AA) entities• Examples (Panned):

– Data Protection• Limits the size, duration, or frequency of a query

– Service Protection• Limits access to functionality of the service

13 – 04/13/23, © 2009 Internet2

Architecture Overview - RP

• The Lookup Service (LS) is a general name for the service and and data discovery infrastructure– Facilitates service and data discovery through the concept of

registration– “Summarizes” and distributes the job of location across layers of

lookup• Home Lookup Services – Local cache of data for several services• Global Lookup Services – Works similar to DNS for locating

information through general queries

• Rolls of the Lookup Service :– Draws specific queries about the data and services away from the

Measurement Points and Archives– Distribute information globally based on local conditions– Assure the ‘freshness’ of information in a dynamic infrastructure

14 – 04/13/23, © 2009 Internet2

Architecture Overview - LS

• The Home Lookup Service (hLS) interacts directly with the other portions of the perfSONAR framework– Recommended deployment is per domain

• Accepts Registration information from around the framework– E.g. An MA will register its name, location, and available metadata– Metadata = static portion of a measurment (‘subject’, not results)

• Responds to Queries about services and data– Services looking for each other (e.g. MP looking for an MA)– Client applications looking for data

15 – 04/13/23, © 2009 Internet2

Architecture Overview - hLS

• The Global Lookup Service (gLS) serves as the oracle of the perfSONAR framework– Global cloud of services cooperating together to distribute

information– Manage the hLSs at the lower layer

• Accepts Registration information from hLSs only(!)– E.g. An hLS will register its name, location, and a summary of the

services and data it contains– Summary = condensed list of domains, ip addresses, data types

• Responds to Queries about services and data– Similar to hLS queries, but more focused on where instead of

what– Answer is typically an hLS to contact, not a direct result

16 – 04/13/23, © 2009 Internet2

Architecture Overview - gLS

17 – 04/13/23, © 2009 Internet2

Architecture Overview – Lookup Service

• Communities = Web 2.0 Content Tagging– Think Flicker (tag your pictures with a category)– Think iTunes (tag your music with a genre)

• How does this help measurement lookup and discovery?– One more axis to search on– More human readable and understandable than IP address

or hostnames• Use as many (or as few) as required:

– Networks (e.g. Campus, Regional, Network)– VO or Project (e.g. USATLAS, eVLBI, etc.)– Organization (DOE)– Other?

18 – 04/13/23, © 2009 Internet2

Architecture Overview - Communities

• Example: Some VO is setting up monitoring.– All sites want to test with each other– Not everyone is coming online at once, and VO membership

may be volatile. – Strategy 1:

• Central VO coordinator maintains a list of participants (and must update it often)

• All monitoring is manual: add/remove test hosts when the list changes

– Strategy 2: • VO recommends a tag for all new hosts• All VO members search for test hosts (periodically) that share

this tag – N.B. the GUIs on the disk can organize this automatically

19 – 04/13/23, © 2009 Internet2

Architecture Overview - Communities

• Screenshot from the toolkit (when setting up the host):

• Top: Communities the host has chosen to associate with• Bottom: ‘Popular’ communities

– The word cloud is based on what we found in the GLS – the larger the word = the more people that are using this classification

20 – 04/13/23, © 2009 Internet2

Architecture Overview - Communities

• Adding hosts via the scheduled testing screen– Manually add– See what hosts are available for a particular community

21 – 04/13/23, © 2009 Internet2

Architecture Overview - Communities

• We are interested in who is involved in the LHC community:

22 – 04/13/23, © 2009 Internet2

Architecture Overview - Communities

• List of hosts from the LHC community:

23 – 04/13/23, © 2009 Internet2

Architecture Overview - Communities

• The Topology Service (TS) gathers and stores network topology information similar to the Lookup Service (LS)– Interfaces with external network tools (Dynamic Circuits, NOC

databases)– Provides a query interface

• Rolls of the Transformation Service :– Gather network topology from various sources– Correlate information found in other TS sources to provide a

complete view of network availability– Interface with measurement tools to associate measurements

with specific portions of the infrastructure

24 – 04/13/23, © 2009 Internet2

Architecture Overview - TS

• The Authentication and Authorization (AS) service serves as a front end for identity management.– Identity management relies on assigning roles to a given user via

attributes, e.g. permission to do something– The AS will communicate via WS with a client and pass along

credentials in order to validate an action or task– The AS will protect access to services and data

• Rolls of the Authentication Service :– Validate services and clients given credentials– Act on behalf of the users to acquire the necessary permissions

25 – 04/13/23, © 2009 Internet2

Architecture Overview - AS

• All communication in perfSONAR is done using well defined XML protocols– Client communication with services

• Informative requests• Measurement Data requests• Discovery (e.g. “tell me where to find X”) queries

– Service to service communication• Where are you type of queries• Registration of data into the LS infrastructure• Self-organizing domains

• The following section outlines the basics of the communication protocol used in perfSONAR– Too much information for a simple slide show – detailed

references available

26 – 04/13/23, © 2009 Internet2

Communication Protocols

• perfSONAR schema and protocol– Understanding the structure of the schema is informative

when discussing the architecture and extensibility mechanisms

• Information Services– The Lookup Service and Topology Service utilize the base

model and provide system “glue” that allows for service discovery and contextualization of both data and services

• Extensibility and ongoing work

27 – 04/13/23, © 2009 Internet2

Architecture and Protocol Interaction

• Key Goals: Extensibility, Normalization, Readability

• Break representation of performance measurements down into basic elements

• Data and Metadata• Measurement Data

– A set of of measurement events that have some value or values at a particular time

• Measurement Metadata– The details about the set of measurement data

28 – 04/13/23, © 2009 Internet2

Architecture and Protocol Interaction

• Can simplify the database representation for many types of measurement data– While optimizations are possible, many

measurement types can be viewed as one value measured over time

• Assists Combination/Concatenation of metrics– Creating derived metrics

• Normalization helps with inferring relationships between types of metrics

29 – 04/13/23, © 2009 Internet2

Schema Normalization

• Subject (Noun)– The measured/tested entity– E.g. An pair of hosts (end-point-pair), or a Layer 3

interface• EventType (Verb)

– What type of measurement, value, or event occurred– Characteristic, tool output, or generic event– E.g. latency, bandwidth, utilization, or simply iperf

• Parameters (Adjectives and Adverbs)– How, or under what conditions, did this event occur?– E.g. buffer sizes used, TCP vs ICMP packets

30 – 04/13/23, © 2009 Internet2

Schema Basic Elements - Metadata

• Some sort of value - Datum– Existence of an event might point to the case

where there no additional value• As in “Link up/down” or threshold events

• Time– Is extensible since various representations are

appropriate in different cases• E.g. UNIX timestamp vs NTP time

31 – 04/13/23, © 2009 Internet2

Schema Basic Elements - Data

MessageMessage

Metadata

Data

32 – 04/13/23, © 2009 Internet2

A Message - Transient Information

Store

Metadata

Data

33 – 04/13/23, © 2009 Internet2

An Object Store – Stationary information

Metadata

<id>someId</id>

Data

<metadataIdRef> someId</metadataIdRef>

34 – 04/13/23, © 2009 Internet2

A Data is Linked to Metadata, via IDs

Metadata

<id>someId</id>

Metadata<id>someOtherId</id>

<metadataIdRef> someId</metadataIdRef>

35 – 04/13/23, © 2009 Internet2

A Metadata can be Linked to Another

• Most elements contain an ID and IDref element– ID = unique identification for this element– IDref = pointer to another unique ID

• IDs have a defined scope– Varies by element type– Ex: Message IDs span a request/response pair– Ex: Metadata/Data ids can be long lived in storage– Check service documentation for specific scope rules

36 – 04/13/23, © 2009 Internet2

IDs

• Metadata can be linked in series in two ways– Merge chaining allows for elements to be reused

and a complete metadata can be built– Operation chaining requests or describes

operations on data sets

AA

BB

ABAB

AA

BB

B(A)

37 – 04/13/23, © 2009 Internet2

Linking Metadata

• Functions applied to the data have URI-based names– Common parameters

• Ideally, a series of these metadata elements can completely describe the provenance of any resultant dataset– As well as requesting selection and reduction

operations at query time

38 – 04/13/23, © 2009 Internet2

Operation Metadata

• A namespace:– http://ggf.org/ns/nmwg/base/2.0/– MAY NOT be a URL

• All measurements have some sort of Data and Time• All measurements can be described by the Metadata

identifying who/what/how• The specific structures of the Data and Metadata

elements depend on the measurement• Approach: Use Data and Metadata elements and vary

the namespaces of the specific elements

39 – 04/13/23, © 2009 Internet2

Schema Namespaces

• We encode the measurement/event type in the namespace– And as a standalone element

• Some components of the system can pass Data and Metadata elements through without understanding their specific structure

• Allows and implementation to decide whether it supports a particular type of data or not

• Allows validation based on extended (namespace-specific) schemata

40 – 04/13/23, © 2009 Internet2

Schema Namespaces

• One key to extensibility is the use of hierarchy with delegation– Similar to OIDs in the IETF management world

• The NM-WG has a hierarchy of network characteristics– Good starting point

• However, not all tools are cleanly mapped onto the Characteristic space– Often a matter of some debate– Can be self defined with an ORG

41 – 04/13/23, © 2009 Internet2

Schema Namespaces & Extensibility

• Organization-rooted tools namespace addresses this

• Some top-level tools• ping, traceroute

• Easy to add new tools in organization-specific namespaces

• Performance Event Repository– Add a schema and get a URI– Add Java classes

42 – 04/13/23, © 2009 Internet2

Schema Namespaces & Extensibility

43 – 04/13/23, © 2009 Internet2

Message Example

• Topology schema grew from network measurement description– Reusable “Subject” elements for common cases

• Also reduces redundancy – Relationships between measurement Subjects

• Same basic structure at all layers– Networks are graphs

• Define:– Node– Port (Interface)– Link– Domain– Network– Path– Service

44 – 04/13/23, © 2009 Internet2

Topology Schema

45 – 04/13/23, © 2009 Internet2

Topology

46 – 04/13/23, © 2009 Internet2

Topology – Recursive Links

• Structured by layers and the same elements recurring there• Varied by namespaces

– Reuse visualization logic, etc.– Validate layer- or technology-specific attributes

• 4 Layers: Base (both abstract and L1), L2, L3, L4• Also technology-specific layers like Ethernet, SONET/SDH• Defining topology is tricky, and lots of related software will

use:– Network Descriptions (static and dynamic)– Measurement tools– Research areas (path finding)

47 – 04/13/23, © 2009 Internet2

Topology Schema

• To completely capture the relationships, we need to do a few more things

• Recursive definition of links– Logical links consist of physical links

• Ordered lists of links - Path– Like above, but we need to introduce an Index attribute

• Networks– Physically consist of links but that is not always the most

convenient logical view– Special element to which Interfaces or Links belong

48 – 04/13/23, © 2009 Internet2

Relationships between Subject

• Elements at the same layer have relationships– A link references two ports/interfaces

• At Layer2 or Layer3

• Elements of the same sort have relationships between themselves at different layers– A Layer 1 Interface (physical NIC) can have one or more Layer 2

Interfaces, which can each have one or more Layer 3 Interfaces

• Node is special – Since a Node doesn’t really have any higher-layer characteristic

independent of its Interfaces

49 – 04/13/23, © 2009 Internet2

Relationships between Subject

• A scheme for identifying network elements• Each network element gets a unique

identifier• This identifier will be included with any

measurement associated with that element.

50 – 04/13/23, © 2009 Internet2

Schema – Network Element Identifiers

• Use Cases:– A topology service can be used to find the

identifier for a network element– An LS could then be queried to find all

measurements associated with that element– Dynamic service path-finding can be integrated

with ongoing measurements

51 – 04/13/23, © 2009 Internet2

Network Element Identifiers

• Identifiers use URN notation– Prefixed with “urn:ogf:network:”– Consists of name/value pairs separated by colons– Possible field names: domain, node, port, link, path,

network– Note: GLIF only recognizes domain= as required,

everything else is opaque– Set of rules defined for each field to keep identifiers

compact and finite

52 – 04/13/23, © 2009 Internet2

Network Element Identifiers

• Examples– urn:ogf:network:domain=Internet2.edu– urn:ogf:network:domain=internet2.edu:node=packrat– urn:ogf:network:domain=internet2.edu:node=rtr.seat:port=so-2%2F1%2F0.16– urn:ogf:network:domain=internet2.edu:node=rtr.seat:port=198.32.8.200– urn:ogf:network:domain=Internet2.edu:node=packrat:port=eth0:link=1– urn:ogf:network:domain=internet2.edu:link=WASH to ATLA OC192– urn:ogf:network:path=anna-11537-176

53 – 04/13/23, © 2009 Internet2

Network Element Identifiers

• Utilizes the basic message container– With type attribute – will vary by service

• Contains various data and metadata elements– Linked via ID or independent

• Uses a message-level parameter element to communicate message-level options– Output format, service annotation

• Messages return a result datum with a type system identical to that of a measurement datum

54 – 04/13/23, © 2009 Internet2

perfSONAR Protocol

• How can I extend perfSONAR?• Definition of metric schema

– If you are publishing a new type of data, schema definition is the first step

• Reuse or re-implement protocol processing– Examples in Java and Perl

• Register with the Lookup Service– Defining a new service type

• Analysis modules are extended by assigning a URI, defining parameters

55 – 04/13/23, © 2009 Internet2

Extending perfSONAR

• The following examples will illustrate the mechanics of services and protocols:– Deploying a Service– Simple Client/Service Interaction

• Echo• Metadata Request• Data Request

– Lookup Service Interaction

56 – 04/13/23, © 2009 Internet2

Framework Interaction

• A perfSONAR service is deployed alongside the measurement infrastructure

• Interactions with the lookup service and clients are described

57 – 04/13/23, © 2009 Internet2

Deploying a Service

58 – 04/13/23, © 2009 Internet2

Deploying a Service

59 – 04/13/23, © 2009 Internet2

Deploying a Service

60 – 04/13/23, © 2009 Internet2

Deploying a Service

61 – 04/13/23, © 2009 Internet2

Deploying a Service

62 – 04/13/23, © 2009 Internet2

Deploying a Service

63 – 04/13/23, © 2009 Internet2

Deploying a Service

64 – 04/13/23, © 2009 Internet2

Deploying a Service

65 – 04/13/23, © 2009 Internet2

Deploying a Service

66 – 04/13/23, © 2009 Internet2

Deploying a Service

67 – 04/13/23, © 2009 Internet2

Deploying a Service

68 – 04/13/23, © 2009 Internet2

Deploying a Service

69 – 04/13/23, © 2009 Internet2

Deploying a Service

70 – 04/13/23, © 2009 Internet2

Deploying a Service

• EchoRequest– Sent to a service to test connectivity– Can be made arbitrarily complex by the service designer

• Test backend storage• Test internal self-checks

– Minimum is an ‘are you alive’ ping

71 – 04/13/23, © 2009 Internet2

Client/Service Interaction

72 – 04/13/23, © 2009 Internet2

Client/Service Interaction

73 – 04/13/23, © 2009 Internet2

Client/Service Interaction

• MetadataKeyRequest– For a given (partial) metadata, ask the service to verify that it does

or does not exist– Return a ‘key’, e.g. replayable token, to access the data

74 – 04/13/23, © 2009 Internet2

Client/Service Interaction

75 – 04/13/23, © 2009 Internet2

Client/Service Interaction

76 – 04/13/23, © 2009 Internet2

Client/Service Interaction

77 – 04/13/23, © 2009 Internet2

Client/Service Interaction

78 – 04/13/23, © 2009 Internet2

Client/Service Interaction

• SetupDataRequest– Given a key or (partial) metadata, return measurement

information.– Can be ‘filtered’ by time to prevent getting more results than

necessary.

79 – 04/13/23, © 2009 Internet2

Client/Service Interaction

80 – 04/13/23, © 2009 Internet2

Client/Service Interaction

81 – 04/13/23, © 2009 Internet2

Client/Service Interaction

82 – 04/13/23, © 2009 Internet2

Client/Service Interaction

83 – 04/13/23, © 2009 Internet2

Client/Service Interaction

84 – 04/13/23, © 2009 Internet2

Client/Service Interaction

85 – 04/13/23, © 2009 Internet2

Client/Service Interaction

• Services register with an hLS• hLSs summarize what they know and pass to the gLSs• gLSs exchange the information as needed• Clients will need a multi-step process to find information

– Query the gLS– Query the appropriate hLS– Query the appropriate services

86 – 04/13/23, © 2009 Internet2

Lookup Service Interaction

87 – 04/13/23, © 2009 Internet2

Lookup Service Interaction

88 – 04/13/23, © 2009 Internet2

Lookup Service Interaction

89 – 04/13/23, © 2009 Internet2

Lookup Service Interaction

90 – 04/13/23, © 2009 Internet2

Lookup Service Interaction

91 – 04/13/23, © 2009 Internet2

Lookup Service Interaction

92 – 04/13/23, © 2009 Internet2

Lookup Service Interaction

93 – 04/13/23, © 2009 Internet2

Lookup Service Interaction

94 – 04/13/23, © 2009 Internet2

Lookup Service Interaction

95 – 04/13/23, © 2009 Internet2

Lookup Service Interaction

96 – 04/13/23, © 2009 Internet2

Lookup Service Interaction

97 – 04/13/23, © 2009 Internet2

Lookup Service Interaction

perfSONAR ArchitectureperfSONAR ArchitectureFebruary 11th 2010, APAN 29 – perfSONAR WorkshopJeff Boote, Senior Network Software Engineer

For more information, visit psps.perfsonar.net

98 – 04/13/23, © 2009 Internet2