DRAFT - ARCFIRE projectict-arcfire.eu/wp-content/uploads/2015/12/arcfire_D4.3.pdf3 Experiment 1: Management of multi-layer converged service provider networks16 ... 6 Experiment 4:

DRAFTGrant Agreement No.: 687871

ARCFIRE

Large-scale RINA Experimentation on FIRE+

Instrument: Research and Innovation ActionThematic Priority: H2020-ICT-2015

D4.3 Design of experimental scenarios, selection of metrics and KPIs

Due date of Deliverable: Month 12Actual submission date: February 17, 2017

Start date of the project: January 1st, 2016. Duration: 24 monthsversion: V1.0

Project funded by the European Commission in the H2020 Programme (2014-2020)

Dissemination level

PU Public X

PP Restricted to other programme participants (including the Commission Services)

RE Restricted to a group specified by the consortium (including the Commission Services)

CO Confidential, only for members of the consortium (including the Commission Services)

DRAFT

D4.3: Design of experimentalscenarios, selection of metricsand KPIs

Document: ARCFIRE D4.3

Date: February 17, 2017

H2020 Grant Agreement No. 687871

Project Name Large-scale RINA Experimentation on FIRE+

Document Name Deliverable 4.3

Document Title Design of experimental scenarios, selection of metrics andKPIs

Workpackage WP4

Authors Dimitri Staessens (imec)

Sander Vrijders (imec)

Eduard Grasa (i2CAT)

Leonardo Bergesio (i2CAT)

Miquel Tarzan (i2CAT)

Bernat Gaston (i2CAT)

Sven van der Meer (LMI)

John Keeney (LMI)

Liam Fallon (LMI)

Vincenzo Maffione (NXW)

Gino Carrozzo (NXW)

Diego Lopez (TID)

John Day (BU)

Editor Dimitri Staessens

Delivery Date December 31st 2016

Version v1.0

2

DRAFT




1 Abstract

This deliverable details the preparatory work done in ARCFIRE WP4 before the actual experimen-tation can begin. It takes the converged network operator scenarios for a traditional network designand the RINA designs that were developed in WP2 together with the testbed overview that wascompiled by T4.1 and extracts interesting reference scenarios for the 4 main experiment objectives.This document will thus serve as a crutch for the experimenters during experimentation, providingall necessary information on the experiment objectives and tools in one location, and providingreferences to the sections of the WP2 deliverables where more details can be found.

First, this document briefly summarises the WP2 reference scenarios for the converged networkoperator, focusing on the architecture of the access network, since that is where the technologydiversification is most apparent. The first experiment will investigate the differences between aRINA and an evolutionary 5G network in terms of management complexity, targeting configurationmanagement deploying a new service. The experiment has been divided into 4 sub-experimentswith green and brown field starting points for the service. The second experiment will performnetwork performance oriented measurements, pitting a 5G scenario based on LTE vs a 5G scenariobased on RINA. Here, ARCFIRE will evaluate network parameters such as overhead with respectto both data transport and mobility management, and scalability with respect to routing, also in thepresence of network failures. The third experiment turns our sight towards multi-provider networks.RINA will be evaluated towards its capability for maintaining end-to-end QoS guarantees withrespect to delay, jitter and bandwidth. Furthermore, it will evaluate how renumbering end-usersaddresses with respect to the location in the network improves overall scalability. Experiment 3will also delve into the OMEC scenario for RINA, keeping applications reachable while they movethrough the network. The fourth and final experiment brings ARCFIRE in the world of DDoSattacks. It will investigate the various checks in RINA’s flow allocation can bar malicious attackersfrom taking down critical services.

3

DRAFT




Table of Contents

1 Abstract 3

2 Reference scenario 102.1 Baseline reference for traditional networks . . . . . . . . . . . . . . . . . . . . . 102.2 RINA network design . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14

3 Experiment 1: Management of multi-layer converged service provider networks 163.1 Objectives . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16

3.1.1 The Current Problem . . . . . . . . . . . . . . . . . . . . . . . . . . . . 163.1.2 Multi-layer Coordination . . . . . . . . . . . . . . . . . . . . . . . . . . 173.1.3 The resulting Objectives . . . . . . . . . . . . . . . . . . . . . . . . . . 183.1.4 Scope: Configuration Management . . . . . . . . . . . . . . . . . . . . 183.1.5 Aspects, KPIs, Scenarios and Use Cases . . . . . . . . . . . . . . . . . . . 21

3.2 Experiment 1-1: Deploy Network from Zero . . . . . . . . . . . . . . . . . . . . 263.2.1 Objectives . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 263.2.2 Metrics and KPIs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 263.2.3 Software . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 273.2.4 Testbeds . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 293.2.5 Experiment scenarios . . . . . . . . . . . . . . . . . . . . . . . . . . . . 293.2.6 Planning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30

3.3 Experiment 1-2: Deploy new Service in Existing Network . . . . . . . . . . . . 303.3.1 Objectives . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 303.3.2 Metrics and KPIs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 313.3.3 Software . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 323.3.4 Testbeds . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 323.3.5 Experiment scenarios . . . . . . . . . . . . . . . . . . . . . . . . . . . . 333.3.6 Planning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33

3.4 Experiment 1-3: Deploy Network and Service from Zero . . . . . . . . . . . . . 333.4.1 Objectives . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 333.4.2 Metrics and KPIs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 343.4.3 Software . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 353.4.4 Testbeds . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 373.4.5 Experiment scenarios . . . . . . . . . . . . . . . . . . . . . . . . . . . . 373.4.6 Planning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38

3.5 Experiment 1-4: Add new Network Node - Zero touch . . . . . . . . . . . . . . 383.5.1 Objectives . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 383.5.2 Metrics and KPIs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 393.5.3 Software . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39

4

DRAFT




3.5.4 Testbeds . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 403.5.5 Experiment scenarios . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 413.5.6 Planning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41

3.6 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41

4 Experiment 2: Deploying resilient, virtualised services over heterogeneous physicalmedia 434.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 434.2 Objectives . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 434.3 Experiment scenario . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 444.4 Metrics and KPIs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 464.5 Software . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 474.6 Testbeds . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 474.7 Planning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48

5 Experiment 3: End to end service provisioning across multiple network providers 515.1 Objectives . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 515.2 Quality of Service levels guaranteed by service provider network . . . . . . . . . . 51

5.2.1 Objectives . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 515.2.2 Metrics and KPIs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 535.2.3 Software . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 535.2.4 Testbeds . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 535.2.5 Experiment scenario . . . . . . . . . . . . . . . . . . . . . . . . . . . . 545.2.6 Planning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 57

5.3 Dynamic and seamless DIF renumbering . . . . . . . . . . . . . . . . . . . . . . 575.3.1 Objectives . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 575.3.2 Metrics and KPIs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 585.3.3 Software . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 595.3.4 Testbeds . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 605.3.5 Experiment scenario . . . . . . . . . . . . . . . . . . . . . . . . . . . . 605.3.6 Planning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 65

5.4 Application discovery, mobility and layer security in support of OMEC . . . . . 655.4.1 Objectives . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 655.4.2 Metrics and KPIs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 665.4.3 Software . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 665.4.4 Testbeds . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 675.4.5 Experiment scenario . . . . . . . . . . . . . . . . . . . . . . . . . . . . 675.4.6 Planning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 71

5

DRAFT




6 Experiment 4: Studying the effects of (Distributed) Denial of Service (D)DoS attacksinside and over RINA networks 736.1 Introduction and motivation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 73

6.1.1 State of the art in botnet formation . . . . . . . . . . . . . . . . . . . . . 736.1.2 State of the art of DDoS attacks . . . . . . . . . . . . . . . . . . . . . . 746.1.3 State of the art of mitigation techniques of DDoS attacks . . . . . . . . . 76

6.2 Objectives . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 786.3 Experiment scenario . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 796.4 Metrics and KPIs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 856.5 Software . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 866.6 Testbeds . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 876.7 Planning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 87

7 Conclusion 89

6

DRAFT




List of Figures

1 Converged service providers’ network . . . . . . . . . . . . . . . . . . . . . . . 102 Residential fixed Internet access: data plane (up) and control plane (down) . . . . . 113 4G Internet access: data plane (up) and control plane (down) . . . . . . . . . . . 124 Layer 3 VPN between customer sites: data plane (up) and control plane (down) . 135 Fixed access network (RINA) . . . . . . . . . . . . . . . . . . . . . . . . . . . . 146 Cellular access network (RINA) . . . . . . . . . . . . . . . . . . . . . . . . . . 147 Enterprise VPNs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 158 Management Points in a Protocol Stack Pattern . . . . . . . . . . . . . . . . . . 169 Management Points in a Layered Domain Pattern . . . . . . . . . . . . . . . . . 1910 FCAPS, Strategies, and RINA Network . . . . . . . . . . . . . . . . . . . . . . 1911 DMS: Stand-alone System for Benchmark Experiments . . . . . . . . . . . . . . 2412 DMS: Full System for Experiments with a RINA Network . . . . . . . . . . . . 2513 Experiment 1: Minimal RINA System for simple Experiments . . . . . . . . . . 2514 Baseline LTE experiment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4515 End-to-end LTE experiment . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4616 Guaranteed QoS levels experiment: abstract layering structure, showing the differ-

ent DIFs to be considered . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5217 Guaranteed QoS levels experiment: abstract layering structure, showing the differ-

ent DIFs to be considered. Provider edge routers are shown . . . . . . . . . . . . 5218 Guaranteed QoS levels experiment: small DIF experiment, backbone DIF connectivity 5419 Guaranteed QoS levels experiment: layering of the scenario . . . . . . . . . . . . 5620 Renumbering experiment: small single DIF scenario, layering structure (up), and

DIF connectivity (down) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6121 Renumbering experiment: large single DIF scenario, DIF connectivity . . . . . . 6222 Renumbering experiment: DIF structure for the multi-DIF scenario . . . . . . . . 6323 Renumbering experiment: systems for small scale multi-DIF scenario . . . . . . 6324 Renumbering experiment: systems for large scale multi-DIF scenario . . . . . . 6425 Categorisation of scenarios and configurations for the renumbering experiments . 6426 OMEC experiment: physical systems involved in the scenario . . . . . . . . . . 6827 OMEC experiment: DIF configurations: UE to server on the public Internet . . . 6928 OMEC experiment: DIF configurations: UE to server on the provider’s cloud . . 7029 Vulnerabilities that enable malware to take control of Internet-connected IoT devices 7430 Classification of DDoS defense mechanisms depending on their deployment . . . 7631 Experiment 4 scenario 1: single top-level DIF (public Internet) configuration . . 8032 Experiment 4 scenario 2: multiple top-level DIFs configuration . . . . . . . . . . 8033 Models for the different DIFs present in the DDoS experiments . . . . . . . . . . . 8134 Systems in the small scale experimental scenario . . . . . . . . . . . . . . . . . 8235 Systems in the large scale experimental scenario . . . . . . . . . . . . . . . . . . 83

7

DRAFT




36 Categorisation of DDoS experimental scenarios . . . . . . . . . . . . . . . . . . 84

8

DRAFT




List of Tables

1 KPIs for experiment 1-1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 282 Software required for experiment 1-1 . . . . . . . . . . . . . . . . . . . . . . . . 293 Milestones for experiment 1-1 . . . . . . . . . . . . . . . . . . . . . . . . . . . 304 KPIs for experiment 1-2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 315 Software required for experiment 1-2 . . . . . . . . . . . . . . . . . . . . . . . . 326 Milestones for experiment 1-2 . . . . . . . . . . . . . . . . . . . . . . . . . . . 347 KPIs for experiment 1-3 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 368 Software required for experiment 1-3 . . . . . . . . . . . . . . . . . . . . . . . . 379 Milestones for experiment 1-3 . . . . . . . . . . . . . . . . . . . . . . . . . . . 3810 KPIs for experiment 1-4 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3911 Software required for experiment 1-4 . . . . . . . . . . . . . . . . . . . . . . . . 4012 Milestones for experiment 1-4 . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4113 KPIs for experiment 2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4714 Software for experiment 2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4815 Testbeds for experiment 2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4816 Planning of experiment 2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4917 KPIs for guaranteed QoS levels experiment . . . . . . . . . . . . . . . . . . . . 5318 Software required for guaranteed QoS levels experiments . . . . . . . . . . . . . 5419 KPIs for guaranteed QoS levels experiments . . . . . . . . . . . . . . . . . . . . 5620 Milestones for guaranteed QoS levels experiment . . . . . . . . . . . . . . . . . 5721 KPIs for renumbering experiments . . . . . . . . . . . . . . . . . . . . . . . . . 5922 Software required for renumbering experiments . . . . . . . . . . . . . . . . . . 5923 Testbeds for renumbering experiments . . . . . . . . . . . . . . . . . . . . . . . 6024 Milestones for renumbering experiments . . . . . . . . . . . . . . . . . . . . . . 6525 KPIs for OMEC experiments . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6626 Software required for OMEC experiments . . . . . . . . . . . . . . . . . . . . . 6727 Testbeds for OMEC experiments . . . . . . . . . . . . . . . . . . . . . . . . . . 6728 Milestones for renumbering experiments . . . . . . . . . . . . . . . . . . . . . . 7229 KPIs for DDoS attacks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8530 Software required for Experiment 4 . . . . . . . . . . . . . . . . . . . . . . . . 8731 Testbeds used in experiment 4 . . . . . . . . . . . . . . . . . . . . . . . . . . . 8832 Milestones for DDoS experiments . . . . . . . . . . . . . . . . . . . . . . . . . 88

9

DRAFT




2 Reference scenario

All experiments in WP4 are grafted in the converged service provider network design as researchedin WP2. This section provides a quick summary of the core designs that will be used in theexperiments. More details can be found in deliverables D2.1 [1] and D2.2 [2].

2.1 Baseline reference for traditional networks

xDSL FTTH WiFi 4G

National DC Regional DC(s)

Metropolitan aggregation


… …

Metro DC

Metro DC

Access

Internet Border

Private peering with other operators , or Internet transit

To Internet eXchange Point (IXP)

IP eXchange border To IPX network (IMS traffic)

micro DC

Metropolitan aggregation Metropolitan

aggregation




Core/backbone

Figure 1: Converged service providers’ network

Figure 1 recaps the main parts of a converged service provider network. The network ispartitioned in three: several types of access networks allow the provider to reach its customersvia wired and wireless technologies. The traffic from these access networks is aggregated bymetropolitan area networks (MANs), which aggregate the traffic towards the network core.

Customers and services are identified and authenticated in the access or MAN. At the core, thetraffic is either forwarded to a DC or an Interconnect edge router (e.g. Internet edge) otherwise.The service provider may have different datacentres attached to different parts of the network:

• Micro data centres attached to the access networks, supporting the Mobile Edge Computingconcept by running very latency-critical services very close to its customers. Micro-data-centers may also be used to support C-RAN (Cloud RAN) deployments.

• Metro data centres attached to the metropolitan networks. These DCs could host serviceproviders’ network services such as DHCP, DNS or authentication servers; but also ContentDelivery Networks (CDNs) or even provide cloud computing services to customers.

• Regional or national data centres attached to the core networks. Could run the sameservices as metro data-centers at a national scale, as well as mobile network gateways and/orthe Network Operations Center (NOC).

10

DRAFT




DSLAM

XDSL

AAL5/ATM

CPE BRAS/Edge router Carrier Eth.

PE Carrier Eth.

PE

802.3

802.1ah

802.1q/p or 802.1ad

Eth PHY Eth PHY

Eth PHY Eth PHY

Carrier Eth. P

802.3

Eth PHY

PPP and PPPoE

Host L2/L1

IP (Internet)

Core router Core router

L2/L1 L2/L1

MPLS (LSP)

Internet Border Router


802.3

Eth PHY

Customer network Service Prov. 2 Service Prov. 1 network

Access

Aggregation Service Edge Core (BGP-free) Internet Edge

Internet eXchange Point Core PoP, city B Core PoP, city A City A Ethernet-based MAN City A Cabinets

MPLS (VPN/Pseudo Wire)

TCP/UDP

DSLAM

XDSL

AAL5/ATM

CPE BRAS/Edge router Carrier Eth.

PE Carrier Eth.

PE

802.3

IEEE 802.1aq (SPB)

802.1q/p or 802.1ad

Eth PHY Eth PHY

Eth PHY Eth PHY

Carrier Eth. P

802.3

Eth PHY

PPP and PPPoE

Host

L2/L1


L2/L1 L2/L1

IP (operator)



802.3

Eth PHY


Access

Aggregation Service Edge Core (BGP-free) Internet Edge

Internet eXchange Point Core PoP, city B Core PoP, city A City A Ethernet-based MAN City A Cabinets

TCP

IS-IS IS-IS IS-IS

iBGP

RSVP (TE) IP

TCP

eBGP

Figure 2: Residential fixed Internet access: data plane (up) and control plane (down)

Figure 2 shows the data and control plane for fixed access. The design shows a Carrier-Ethernetbased MAN aggregating traffic and forwarding it to one or more BRAS routers located at a corePoP. In the control plane, a BRAS (connected over PPP) authenticates customers. Shortest PathBridging (SPB) in the MAN enables traffic engineering. eBGP is used by the provider border routerto exchange traffic with its peer or upstream routers. Routes are disseminated to the BRAS(es) viaiBGP. The BGP-free core runs an MPLS control plane.

11

DRAFT




DSLAM eNodeB

Multi Service Edge Carrier Eth.

PE Carrier Eth.

PE

802.1ah

802.1q/p or 802.1ad

Eth PHY Eth PHY

Eth PHY Eth PHY

Carrier Eth. P

MAC

PHY

UE

802.3

Eth PHY

P-GW

Service Prov. 1 network

Access Core (BGP-free) Internet Edge

Core PoP, city B Core PoP, city A

City A Ethernet-based MAN City A Antenna sites

RLC

PDCP

S-GW

IP (provider)

UDP

GTP-U

Eth PHY

802.3

UDP

GTP-U

IP (Internet)

Aggregation

L2/L1


L2/L1 L2/L1

MPLS (LSP)



802.3

Eth PHY

Service Edge Mobile gateways

DSLAM

eNodeB Multi Service Edge Carrier Eth.

PE Carrier Eth.

PE

802.1aq (SPB)

802.1q/p or 802.1ad

Eth PHY Eth PHY

Eth PHY Eth PHY

Carrier Eth. P

MAC

PHY

UE 802.3

Eth PHY

P-GW

Service Prov. 1 network

Access Core (BGP-free) Internet Edge

Core PoP, city B Core PoP, city A

City A Ethernet-based MAN City A Antenna sites

RLC

PDCP

S-GW

IP (provider)

Eth PHY

802.3

UDP

GTP-C

Aggregation Service Edge Mobile gateways

RRC

802.3

Eth PHY

MME

S1-AP

NAS

SCTP GTP-C

UDP

L2/L1


L2/L1 L2/L1

IP (operator)


802.3

Eth PHY

TCP

IS-IS IS-IS IS-IS

iBGP

RSVP (TE) IP

TCP

eBGP

Figure 3: 4G Internet access: data plane (up) and control plane (down)

Figure 3 shows the data (user) plane and control plane for wireless (4G) access. The eNodeB(base station) is attached to the aggregation network, which aggregates the traffic from multiplebase stations into a Core PoP. The core PoP contains a Multi Service Edge and forwards the trafficto the mobile network gateways forming the EPC (Evolved Packet Core). In the example, the S-GWand the P-GW are located at the core PoP. Internet traffic reaching the P-GW is forwarded to one ofthe provider’s Internet border routers through the core network. In the control plane. the UE runsthe NAS protocol against the Mobility Management Element (MME), which allows the MME toauthenticate the user and negotiate handovers. The P-GW runs iBGP with Internet border routers,allowing it to route UE traffic to the Internet.

12

DRAFT




CPE MS Edge router

Carrier Eth. PE

Carrier Eth. PE

802.1ah

802.1q/p or 802.1ad

Eth PHY

Eth PHY Eth PHY Eth PHY

Carrier Eth. P

802.3

Eth PHY

Host L2/L1

IP (VPN)


L2/L1 L2/L1

MPLS (LSP)

MS Edge Router

Customer network Service Prov. 1 network

Aggregation Service Edge Core (BGP-free)

Core PoP, city B Core PoP, city A City A Ethernet-based MAN


TCP/UDP

Carrier Eth. PE

Carrier Eth. PE

802.1ah

Eth PHY Eth PHY

Carrier Eth. P

802.1q/p or 802.1ad

Eth PHY

802.3

Eth PHY Eth PHY

Aggregation Service Edge

Customer network

City B Ethernet-based MAN

Host

CPE

CPE

MS Edge router

Carrier Eth. PE

Carrier Eth. PE

802.1aq (SPB)

802.1q/p or 802.1ad

Eth PHY

Eth PHY Eth PHY Eth PHY

Carrier Eth. P

Core router Core router MS Edge Router

Customer network Service Prov. 1 network

Aggregation Service Edge Core (BGP-free)

Core PoP, city B Core PoP, city A City A Ethernet-based MAN

Carrier Eth. PE

Carrier Eth. PE

802.1aq (SPB)

Eth PHY Eth PHY

Carrier Eth. P

802.1q/p or 802.1ad

Eth PHY Eth PHY

Aggregation Service Edge

Customer network

City B Ethernet-based MAN

CPE L2/L1 L2/L1 L2/L1

IP (operator)

TCP

IS-IS IS-IS IS-IS

iBGP

RSVP (TE)

IP

TCP

eBGP

IP

TCP

eBGP

Figure 4: Layer 3 VPN between customer sites: data plane (up) and control plane (down)

Figure 4 illustrates data and control plane for a Layer 3 VPN service between two businesssites. The customer CPE router is directly attached to the MAN. The L3 VPN service is carriedover a VLAN to the MS Edge router, which is running a Virtual Routing and Forwarding (VRF)instance to forward the VPN traffic through the MPLS core. When VPN traffic exits the MPLScore, another VRF instance forwards it through another VLAN over the MAN delivering it to theCPE at the other business site. The control plane runs eBGP between the CPE and the MS Edgerouters to exchange VPN route information. These VPN routes are disseminated over the core viaiBGP, allowing all VPN locations to learn the routes required to forward the VPN traffic across allsites.

13

DRAFT




2.2 RINA network design

This section summarises the RINA network design for the 3 scenarios above.

Host

CPE

PtP DIF

Home DIF

Public Internet or VPN or app-specific DIF

Access Router

Edge Service Router

PtP DIF . . . PtP DIF

Host

Service provider top level DIF

PtP DIF Aggregation

DIFs

Customer network

Provider network

Local loop Aggregation

. . .

Figure 5: Fixed access network (RINA)

Figure 5 shows a RINA network structure for a fixed access network. The CPE is connected tothe access router via a point-to-point DIF. This DIF provides IPC services to one or more serviceDIFs, which are used to authenticate the customer and support access to one or more utility DIFs,such as a public Internet DIF or VPN or application-specific DIFs. The traffic from multiple accessrouters is multiplexed over the aggregation network into an edge service router, which forwards itfurther towards its final destination.

IP (e.g. Internet)

TCP or UDP

PDCP GTP-U

Protocol conversion

GTP-U

RLC

MAC

L1

UDP

IP (LTE transport)

MAC MAC . . .

L1 . . . L1

UDP

IP (LTE transport)

MAC MAC . . .

L1 . . . L1 UE

eNodeB S-GW P-GW

EPS bearer EPS bearer

LTE-Uu

S1-U S5/S8

MAC

L1

SGi

Public Internet DIF

Radio DIF . . .

. . . . . . UE

eNodeB S-GW P-GW

LTE-Uu

S1-U S5/S8 SGi

PtPDIF

Metro DIF

PtPDIF PtPDIF

PtPDIFMobile Network Top Level DIF

Backbone DIF

App

App

PtPDIF

Figure 6: Cellular access network (RINA)

Figure 6 shows a possible RINA network for mobile access. A radio multi-access DIF managesthe radio resource allocation and provides IPC over the wireless medium. A Mobile networktop-level DIF provides flows spanning the scope of the mobile network, where the Metro and

14

DRAFT




Backbone DIFs multiplex and transport the traffic of the Mobile Network Top-Level DIF. Finally,the public Internet DIF allows applications in the UE to connect to other applications available onthe public Internet.

PE-rs 1

IPCP

PE-rs 2 P2 P1 CE 2 CE 1 H1 H2

Customer 1 Site 1 Network

Service Provider Network


MTU-s 1

IPCP IPCP

MTU-s 2

IPCP IPCP

P (metro) P (metro)

IPCP

App App

PtP DIF

Metro DIF Backbone DIF Metro DIF

Green VPN DIF

VPN Service DIF

PtP DIF PtP DIF PtP DIF PtP DIF PtP DIF

PtP DIF

PtP DIF

PtP DIF PtP DIF PtP DIF

PE-rs 1

VPLS

PE-rs 2 P2 P1 CE 2 CE 1 H1 H2


Service Provider Network


MTU-s 1

VPLS VPLS

MTU-s 2

VPLS VPLS

P (metro) P (metro)

VPLS

App App TCP

IP

802.3

Eth/Net Eth/Net

MPLS

MPLS

Eth/Net Eth/Net Eth/Net

MPLS

MPLS

Eth/Net Eth/Net

MPLS

MPLS

802.3 802.1q

PBB (802.1ah)

Protocol conversion

Figure 7: Enterprise VPNs

Figure 7 shows a RINA configuration for providing Enterprise VPN services. The variousrouters are connected over various Point-to-Point DIFs. The Metro DIF provides IPC over metroaggregation networks, multiplexing the traffic of the different types of services the operator providesover the metropolitan segment. A backbone DIF provides IPC over the core segment of the network,interconnecting PoP in different cities. A VPN service DIF on top of the Metro and Core DIFallocates resources to different VPN DIFs over the entire scope of the provider network.

15

DRAFT




3 Experiment 1: Management of multi-layer converged service pro-vider networks

3.1 Objectives

The main goal of experiment 1 is to demonstrate the difference between a (protocol) stack and aDIF. Traditionally, in the literature as well as in commercial environments, the network is seen asa stack or protocol stack, realised by all components in the network. Figure 8 shows an exampleof this view using the resource abstraction introduced in [2]. Each vertical in the figure showsa network component, for instance an end system, a border router, or an interior router. Eachpair of communication components must implement entire stacks that are compatible with eachother, where most components must implement the whole stack (left and right) or parts of thestack (middle) depending on what network functionality it provides. In the figure, the left and rightcomponent could be end systems and the middle component could be a switch or router.

Layer n

Layer n+1

Layer n-1

Figure 8: Management Points in a Protocol Stack Pattern

3.1.1 The Current Problem

The management of stacks is complex because it is necessary to coordinate vertical as well ashorizontal aspects, each separated per component and provided stack element. On the horizontalaspect, each part of a communication service (each stack element on the same layer in eachcomponent), needs to be configured individually. A reconfiguration often requires to reconfigure allstack elements of all components. This situation is even more complicated when we apply a real

16

DRAFT




word scenario, in which the ownership of the components varies. Any initial configuration and anyfurther reconfiguration must now be coordinated amongst all components and their respective owner(assuming that the owner has administrative permissions to perform the configuration action).

On the vertical, the configuration of each stack element must be configured in a way that thecomponent overall can provide the advertised and required service. Here, ownership is not theproblem. Instead the multi-vendor, multi-protocol, and multi-technology aspect of each individualstack layer provides for extensive complexity.

In a 5G converged service provider network, we must also consider a separation of technologyand ownership across the whole network, for components (here called nodes) as well as the“protocol zoo” dealing with various aspects of the network (and service) operation. Radio nodesform part of the radio network (RAN). Nodes aggregating radio access traffic and managing usersessions are found in the core network. Long distance communication and links to the publicInternet belong to the transport network. Each of those networks is often owned by different partsof a provider’s organisation.

This mix makes coordinated management of (protocol stacks) virtually impossible, mainlybecause the state space that needs to be coordinated is too large. The current solution is twofold.First, a large amount of standards is produced as normative source for management activities (theactivities themselves as well as for the underlying stack). Those standards provide interoperabilitybut limit flexibility. Second, management is often based on management models using managedobjects (with standardised access in form of a defined protocol and standardised information models,often in terms of a management information base). A large number of models exists and needs tobe considered for management activities. The figure 8 also shows the management points (blueinterfaces) and the internal management or configuration policies that need to be coordinated. Whilethe figure suggests a largely normalised environment, reality is much more heterogeneous.

3.1.2 Multi-layer Coordination

In RINA, the situation is very different. Using a common API (the IPC API) and building immutableinfrastructure used by each and every IPCP means that there are no longer an technologicaldifferences among different layers in the network. There are also no differences between verticalelements in a component, everything is an IPC process (IPCP).

Next, the concept of separation of mechanism and policy, and the definition of policies forall aspects of an IPCP (and thus inter-process communication), makes all available policies ex-plicit. There is no need anymore for separate management model interface definitions, everythingmanagement requires is already defined by the architecture.

Next, the concept of a DIF as a layer, and not a stack, with autonomic functions for layermanagement (as part of every IPCP) and with an explicit definition of the scope of the layer (thescope of the DIF, realised by its policies), provides for a very new concept with regard to multi-layermanagement.

Last but not least, the RIB in the IPCPs (and thus in the DIF) provides for the required

17

DRAFT




information base for all of the above. The RIB maintains all shared state of all IPCPs in a DIF. Thedefined processes also realise an automatic update of this shared state amongst all IPCPs in a DIF.This means that changes to a DIF’s configuration will propagate from one IPCP to another until theentire DIF is reconfigured.

3.1.3 The resulting Objectives

The objectives of experiment 1 are

1. to demonstrate that the simplification RINA provides (extensively discussed in [2] andsummarised above) lead to a significant simplification of the required management definitionsand implementations,

2. to demonstrate that, since the manager can use the same model with a few variations based ondifferent RINA policies (DIF policies) to configure multiple layers, coordinated managementcan evolve from a complex management task (often provided by workflows with associatedmanagement policies) into a strategy-oriented management task in which the managementactivities are solely described by management strategies (as discussed in [2]), and

3. to demonstrate that the manager can perform an adequate number of strategies at the sametime (single manager deployment), and

4. to demonstrate that the manager can be scaled up and scaled down in case a single managerdeployment is not able to handle the load of executed strategies

Multi-layer management can now evolve from stack management (vertical and horizontal)towards an actual coordination function in a multi-domain model (here the domains are DIFs).Figure 9 shows this new situation provided by RINA using the resource abstraction introduced in[2], including the management points (per domain element - IPCP, per domain - DIF, multi-domain,and for the management system on the right). Most of the control functions are realised in theIPCPs and the DIFs. DIF interaction is also provided by RINA. What is left for the coordination isto provide for a coordinated initial configuration, and later for possible re-configuration as part ofthe management activity (monitor and repair, see [2].

3.1.4 Scope: Configuration Management

Management activities cover a wide range of functional aspects. Since this experiment cannot coverall aspects, we will narrow the scope to a representative set of those aspects.

Network management activities are often called management functions. The functional scope ofthose management functions is commonly described in terms of Fault, Configuration, Accounting,Performance, and Security (FCAPS), also known as management areas [3]. On top of network

18

DRAFT




Domainn

Domainn-1

Domainn+1

Mul�-Domain

Figure 9: Management Points in a Layered Domain Pattern

management (including layer management) system management functions are standardised tomanage a single system, see for instance ITU-T recommendations X.730 to X.799.

A management system, including the DMS, follows the common manager-agent paradigm,where a manager executes the management functions and a managed agent controls resources. Thecommunication between manager and agent is then the management protocol. Communication isrestricted to operations from manager to agent and notification from agent to manager. The agentuses a Management Information Base (MIB) as a standardised knowledge base of the resources itcontrols. The resources commonly provide a standardised interface to the agent.

Fault

Strategy

Strategy

Strategy

Strategy

CDAP

Opera�ons

No�fica�ons

Manager ManagementAgent

Distributed IPC Facility (DIF)

Border RouterInterior Router

Host Host

App A App B

DIF DIF

DIFDIF

Border Router

DIF

IPCPIPCP

IPCP

IPCP

IPCPIPCPIPCPIPCPIPCP

IPCP

IPCP IPCPIPCPIPCP

IPCP

Configura�on

Performance

Accoun�ng

Security

Figure 10: FCAPS, Strategies, and RINA Network

Figure 10 shows an example of the manager/agent paradigm in a RINA context. On the managerside, the FCAPS management areas are used to define the management functions. Strategies realisethose functions, so they are the management activities. The management protocol is CDAP. TheManagement Agent (MA) then controls a RINA network, i.e. a deployment of DIFs with IPC

19

DRAFT




processes. The MIB in RINA is superset of all the RINA Information Bases (RIB) of the controlledIPC processes. The common (standardised) management interface of the resources (DIFs, IPCprocesses) is the common IPC API.

As stated above, this experiment cannot cover all functional areas. For the FCAPS, the followingassumptions can be made to narrow the scope for the experiment:

• Security - the management of security aspects is covered in the ARCFIRE experiment 4.

• Accounting - while the collection of data for resource use can be realised. Since a DIF has awell-defined scope, such a collection can be done per DIF (and then per IPC process in theDIF) with the full understanding of the DIFs scope. Call Data Records (CDRs) or other baseinformation for accounting can then be produced. However, to demonstrate full AccountingManagement (AM), we would need to associate the usage data (e.g. CDRs) to user session,then individual users, and finally to the contract those users have with the service provider.This means building such a system requires a significant amount of resources outside theoriginal scope of the project yet with (very likely) little added value for the experiment.

• Performance - collection of performance data is very similar to collection of accountingdata. However, to demonstrate full Performance Management (PM) requires evaluating thecollected performance counters against the KPIs a network operator (or service provider) setfor its operation. Those goals differ from operator to operator and from provider to provider.They also depend on the optimisation strategy for a network (for instance optimise for voice,or data, or coverage, or against connection loss / call drop, etc.). There is little discussion inthe literature on common performance KPIs, beside some higher level goals such as optimalperformance. Since general performance tests of a RINA network have been done in otherprojects, performance management for this experiment will not add much value (besidesthe resource heavy and largely arbitrary design and implementation of several optimisationstrategies to test).

• Fault - faults are problems in the network that can be observed either directly (the resource andthen the Management Agent send an error in form of a notification) or indirectly (by observingthe behaviour of the network comparing it to the anticipated or required behaviour). Anexample of a directly observed fault is the loss of connectivity due to a hardware problem (portbroken, physical node down). Examples for an indirectly observed fault is congestion control(monitor the network for indicators of congestion) or the classic sleeping cell scenario (aradio node is alive, not down or broken, but does not accept any further traffic or connectionsfrom mobile equipment). While Fault Management (FM) is often seen as the most importantmanagement activity, it is very hard to define fault scenarios for a repeatable experimentationwith measurable KPIs. On one side, we can of course design an experiment in which anIPC process is deliberately broken, creating a fault, resulting in the DMS executing a faultmitigation strategy. On the other side, RINA is an autonomic network, that realises much

20

DRAFT




control (and management functionality, in the IPC process layer management) locally withoutnotifying the DMS. The actual possible fault scenarios in such an environment have not yetbeen studied in detail, thus making it very hard to design fault management scenarios.

• Configuration - every network operation starts with planning (design a network for givenrequirements) followed by deployment (of network nodes and other physical resources andsoftware components and other virtual resources). Once in operation, a network might(and usually does) require re-configuration. This requirement can be the result of a changein requirements (up to business goals set by the operator), faults in the network (faultswhich can be mitigated by re-configuration), a required shift in the operation due to changedtraffic (traffic behaviour or traffic mix impeding for instance required performance), theintroduction of new accounting measurements required for new accounting (and finallycharging) strategies, or changes for the security of the network operation (or parts of it).From this viewpoint, configuration management is the facilitator of all other functionalmanagement areas. Thus the main target for this experiment is Configuration Management(CM). Another reason to focus on CM is the current situation of support for deploying aRINA network. The RINA Demonstrator is the most efficient way to define a RINA networkand deploy it. The Demonstrator automates a large amount of the underlying configurationspecification and the deployment actions. In defining strategies for CM for this experiment,we can (i) benefit from the developed Demonstrator (specification requirements and workflowhave been developed, tested, and are in use for demonstrations for a long time) and (ii)provide a simpler, more dynamic, and more flexible solution than the demonstrator is today.A DMS with a set of CM strategies can supplement the Demonstrator and help to increasethe automated deployment of RINA networks. Adding reconfiguration (in the DMS) willfurther enhance demo capabilities.

3.1.5 Aspects, KPIs, Scenarios and Use Cases

Measuring a network management system for performance, cost, and quality is a very difficult task.All three aspects depend on many variables, e.g. network topology, network size, operational opti-misation strategies, available resources (compute, storage, network) for the network managementsystem, different deployment options (e.g. clustering) of the network management system, andmore. Beside measurable technical KPIs, a network management system has to support KPIs thatdepend on operator-specific goals and metrics, because a network management system will be usedin a specific environment of a particular operator, often a Network Operation Centre (NOC). SomeKPIs used are:

• network server availability

• server (NMS) to administration (NOC personal) ratio

21

DRAFT




• network availability

• capacity of network path (to and from the NMS and in the managed network)

• Critical outage time (how long does it take the NMS to mitigate critical outages (i) in thenetwork and (ii) for itself)

• Frequency of monitoring checks (how often is the NMS required to pull monitoring informa-tion for its own operation)

• Number of affected users (NOC users) on an average basis, i.e. how many people are requiredto use the NMS to operate a network

• Indices of smoothness of network changes, service introduction, etc.

• Complexity of network management operations, e.g. how many different actions need to beinvoked to solve a problem

• Performance of network management operations, e.g. how many similar actions are need tobe invoked solve a problem

• Workflow complexity, e.g. how complex are workflows for problem solving strategies

• Touchpoints, e.g. how many touchpoints are required to realise a network managementoperation (zero-touch here means none, as the ultimate goal and key indicator for a highdegree of automation in the network management system)

Detailed statistics and performance of Operation Support Systems (OSS) used by mobileoperators cannot be obtained. This information is often considered to be privileged by operators aswell as OSS vendors. The same situation applies to metrics on NMS ease of use, involved personaland other costs, and quality.

The closet equivalent to the ARCFIRE DMS in the real world are SNMP based managementsystem for mobile, core, and transport networks as well as for LAN/MAN deployments. However,as [4] states, the performance of SNMP (and related NMSs) has never been fully and properlystudied. There is no commonly agreed measurement methodology to assess network management(and SNMP) performance). In addition, similar to OSS, little is known about SNMP (and SNMP-based NMS) usage patterns in operational networks, which impedes design of realistic scenariosfor an NMS analysis.

The search for relevant, measurable, and comparable KPIs for this experiment is furthercomplicated by the radical new value propositions of a RINA network. For the first time we have afully autonomic network available, including explicit specifications of the scope of layers (DIFs),automatic mechanisms for sharing state between layer elements (the IPC process with their RIBinside a DIF), fully policy-based local control (data transfer and data transfer control), and the

22

DRAFT




availability of a complete set of layer management functions (the layer management functions insidean IPC process). Multi-layer (multi-DIF) coordination now becomes a very different and simplertask (compared to current OSS and NMS systems). Even comparing it to cross-layer optimisationis difficult, since this optimisation assumes a role of control over the layers while the ARCFIREDMS operates in terms of monitoring and repair. The starting point of management activities isvery different, making comparisons extremely difficult (or arbitrary).

To provide a consistent, measurable, and comparable set of performance indicators for the DMSwe will focus on three aspects:

• Speed: the speed of management operations can be measured by the number of collected,required, and adjusted variables (policies and configurations of DIFs); the number of notifica-tions for a management activity; and experienced delays (in the management activity).

• Cost: can be measured by CPU usage (or wider usage of compute resources in a cloudenvironment), memory usage (or the usage of storage resources in a wider cloud environment),and bandwidth usage (how much overhead in required bandwidth does management create).

• Quality: quality can be measured in a spatial-temporal context. Spatial here refers todifferences in particular variables (configuration and policies) between the DMS and theRINA networks. Temporal refers to errors due to temporal aspects, such as monitoringroundtrip times. Loss can be added to evaluate if information is lost in transport.

For these three aspects, we can now look into relevant KPIs. We have identified the followinggeneral KPIs for this experiment:

• Speed: the speed of management strategies

• Scale: the required scale of the DMS for speedy operation

• Time and cost for scale-in and scale-out: how timely can the DMS be scaled out (extended)in case of cascading event storms, how costly is this change of scale

• Touch: how many touches are required for management strategies to succeed (this can beused as an indicator for cost since it determines the extent of human input required for anoperation)

• Complexity: how complex are the used management strategies, i.e. how many differentoperations do they require (internally to make decisions and externally to realise them)

• Degree of automation: using the touch KPI we can estimate the degree of automation theDMS will provide in a network operation

23

DRAFT




Aspects and general KPIs need to be measured in experiments. Those experiments can beclassified as a combination of a scenario with a use case. There are many possible scenarios anduse cases to select from. The following scenarios are considered:

• No network: this scenario can be used to benchmark the DMS and all developed strategies.Strategy triggers (and responses) need to be simulated but actions are not performed. Thefocus is on the DMS performance only. The DMS configuration is shown in Figure 11.

• Small (minimal) RINA network: a simple deployment of two hosts, each connected to adedicated border router, both border routers connected to the same interior router. The DMSconfiguration is shown in Figure 12. The RINA network setup is shown in Figure 13.

• Medium RINA network: a medium size RINA network as for instance used in the renumber-ing demonstration (30 nodes to show a network on European scale as used in experiment 3,cf Figure 20) or the data center demonstration (38 nodes for a medium size data center usinga spine-leaf configuration). The DMS configuration is shown in Figure 12.

• Large RINA network: a large RINA network like the network used in the experimentresembling the AT&T transport network for managed services (as used in experiment 3, cfFigure 21) The DMS configuration is shown in Figure 12.

DMS

DMSStrategy Engine

RIB

ManagementShell / GUI

OperatorOSS/NMS and

OpenNMS

Message Bus (W3C WebSocket)

DMSStrategy Engine

Figure 11: DMS: Stand-alone System for Benchmark Experiments

Each these scenarios can be combined with each of the following use cases:

• Deploy network from zero: no RINA network exists, the DMS is triggered by an operatormechanism and deploys a complete RINA network.

• Deploy a new service (DIF) in existing network: A RINA network exists, the DMS managesit, now add services and applications to the network by creating a new DIF (service) anddeploying applications.

24

DRAFT




RINA

DMS

ManagedResource

(RINA)

ManagementAgent(MA)

DMSStrategy Engine

API Calls, etc.CDAP

RIB RIB RIBSynchroniza�on Synchroniza�on

ManagementShell / GUI

OperatorOSS/NMS and

OpenNMS

Message Bus (W3C WebSocket) CDAPConnector

DMSStrategy Engine

Figure 12: DMS: Full System for Experiments with a RINA Network

Distributed IPC Facility (DIF)

Border RouterInterior Router

Host Host

App A App B

DIF DIF

DIFDIF

Border Router

DIF

IPCPIPCP

IPCP

IPCP

IPCPIPCPIPCPIPCPIPCP

IPCP

IPCP IPCPIPCPIPCP

IPCP

Figure 13: Experiment 1: Minimal RINA System for simple Experiments

25

DRAFT




• Deploy network and service from zero: Deploy the network nodes, all DIFs, and an applica-tion.

• Add a new node to network: how many touch points are required to add a new node, e.g. is itpossible to create a zero touch strategy for adding new nodes?

The resulting 4-tuple for this experiment then is aspects, general KPIs, scenarios, and use cases.For concrete experiments we should define concrete KPIs and design the required strategies. It isalso clear that a separation into sub-experiments is required. This separation can be done usingthe scenarios or the use cases. We have decided to take the use cases as key separator and run allscenarios in each use case.

The following sub sections detail the resulting 4 sub experiments. Each sub experimentaddresses one use case for all scenarios using all KPIs covering all aspects. Therefore, the subexperiment description is in parts repetitive. We chose to leave the repetition in this documentation,because different teams may only work a subset of the sub experiments.

3.2 Experiment 1-1: Deploy Network from Zero

3.2.1 Objectives

This experiment assesses the DMS’s capability to build a network from scratch using the specifica-tion of an operator’s network planning, e.g. topology and required network services as input. At thestart, there is no network, i.e. no network node is up and running. The first step is to start the firstnetwork node. Each following step adds a new network node according to the given topology.

The configurations of all nodes is handled by the DMS. A single strategy should be specified,which takes a topology (and all required information for specific nodes, e.g. required services andDIFs) and once triggered, builds the network step by step (i.e. node by node).

The experiment is finalised by a second strategy, which deploys a number of tests in the newlybuild RINA network to evaluate the correctness of the configuration. Those tests can be active(deploy a test service and evaluate correctness) or passive (analyse the configuration of each nodeas represented in the RIB against the given topology).

The strategy which builds the network can be configured for the three RINA network scenarios.The experiment can then be run for the benchmark scenario and all three RINA network scenarios.We do not anticipate different strategies for the different scenarios.

The initial trigger for building the RINA network comes from the operator’s OSS/NMS. Thiswill be simulated by a skeleton OSS/NMS trigger application in the DMS. Evaluation of the builtnetwork in the benchmark scenario can only be performed passively.

3.2.2 Metrics and KPIs

This experiment uses the KPIs introduced in section 3.1.5 in terms of speed, scale, time and cost,touch, complexity (of the strategy), and degree of automation.

26

DRAFT




They are further specialised for this experiment as shows in Table 1. This experiment shouldquantify:

• the speed a node is added to the network (the complete transaction from strategy trigger towhen the node is up and running)

• the speed a whole RINA network is built (the complete transaction from strategy triggerto when the network is up and running), very likely determined by the number of similaroperations the management strategy must realise

• the scale of the required DMS (one or more executors, possibility of parallelising the actionsfor adding nodes)

• time and cost to scale the DMS up or down (e.g. per RINA network scenario or when runningdifferent scenarios in sequence)

• the complexity of the management strategy, i.e. the number of different operations thestrategy must realise

• touches and degree of automation, evaluated as the number of touches required to build acomplete RINA network

3.2.3 Software

This experiment will require several software items for its execution, summarised in Table 2.

• IRATI RINA implementation. It will be used to implement multiple RINA-enabled systemsand run a variety of DIFs according to the experiment configuration.

• rina-echo-time. The rina-echo-time application will be used to measure the latency andpacket loss perceived by a flow user while the experiment is running.

• rina-tgen. The rina-tgen application will be used to create traffic while the experiment isrunning.

• Demonstrator. The Demonstrator will be used to provide as a base tool for the configurationof the RINA network. Once an associated strategy is developed, it should be possible toremove the Demonstrator from the experiment.

• Rumba. Rumba is the experimentation and measurement framework tailored to RINAexperiments. It allows the definition of complex experiments involving multiple testbeds,being able to interact with testbed federation/automation tools such as jFed.

27

DRAFT




KPI Metric Current state of the art ARCFIRE Objective

E1-1.1 Speedy node add ms or s Node creation and config-uration is often realised byrather complex workflowsthat can take several minutesup to an hour (for automatedcreation)

the strategy should run insub-second speed for singlenode creation

E1-1.2 Speedy networkcreation

ms or s or min Creation and configurationof a whole network can,depending on the networkcomplexity, take severalhours to several weeks

Even the large RINA net-work should be created inminutes, rather than hours

E1-1.3 DMS scale Number of strategy execu-tors

Multiple workflows executedin parallel (or realised byteams working in parallel)

Ideally only 1 executorrequired (sufficient speed),more (with parallelisedexecution) if speed KPIcannot be met

E1-1.4 DMS Scale Out s and CPU usage Scaling out a managementsystem that is in operationis often impossible or ex-tremely costly (takes hours,requires multiple touches, isprocessing intensive)

Scaling out the DMS shouldbe done in seconds, with 1or zero touch, and minimalCPU costs (for the scalingitself)

E1-1.5 Strategy Complexity number of different opera-tions required

Currently, CM operationscreate large sets of CMprofiles, one per node

From early experiments, weestimate 4-5 different oper-ations per node replicatedfor each node to create thenetwork

E1-1.6 Touches / Degree ofAutomation

number of touches Beside the (not counted)original touch (installationof hardware node or triggerfor software installation),adding a node often requiresmultiple additional touchesfor configuration. Early pro-totypes for VNF deploymentcan operate on a minimum(potentially zero) touchbasis.

Ideally zero touch

Table 1: KPIs for experiment 1-1

28

DRAFT




Software Description License

IRATI [5] RINA Implementation for Linux OS GPL and LGPL

rina-echo-time [5] Measure latency and packet loss GPL

rina-tgen [6] Traffic generator, measure goodput GEANT OS License

Demonstrator [7] Tool to setup self-contained VM-based IRATI testbeds GPL

Rumba [8] Tool to setup RINA implementation experiments on FED4FIRE+ testbeds LGPL

DMS Management system with strategy executor, OSS/NMS trigger, management shell,event visualiser

TBD

Table 2: Software required for experiment 1-1

• DMS The DMS is the core system under evaluation in the experiment. It will be deployedin the required configuration (see experiment scenarios). The development of an associatedstrategy for the experiment is important, which will be deployed in the DMS and triggered bythe experiment. An additional small application will be used in the DMS to trigger strategyexecution, in particular an OSS/NMS trigger simulator for the benchmark scenario.

3.2.4 Testbeds

The Virtual Wall will be the main testbed of reference as reported in D4.2 [9]. The Virtual Wallcan provide access to bare metal hardware to run the RINA implementation and the applicationsrequired for the experiment, providing access to an adequate number of resources.

Initially the jFed experimenter GUI will be used to reserve and setup the resources in theFED4FIRE+ testbeds. Once it becomes available, the experimentation and measurement frameworkunder development in WP3 will provide a more automated front-end for jFed as well as other IRATIdeployment and configuration tools such as the Demonstrator.

The DMS will be deployed along with the IRATI Demonstrator.In summary, this experiment may use the following testbeds from the selection made in D4.2

[9]:

Testbed Purpose

Virtual Wall Provides access to enough bare metal servers to carry out experiment 1-1

3.2.5 Experiment scenarios

The experiment will be run in four different scenarios.No RINA Network: This scenario uses triggers from the operators OSS/NMS (here simulated

by a skeleton application) to benchmark the DMS strategy execution. All developed strategies willbe tested for speed and scale. The DMS will be a stand-alone DMS as shown in Figure 11.

29

DRAFT




Milestone Month Description

MS1 M16 DMS Software with Strategy Executor, OSS/NMS trigger, MA/Demonstrator Integration

MS2 M17 Strategy for experiment defined and tested

MS3 M21 reference experiment with measurements on LMI reference server for direct comparison for all scenar-ios

MS4 M22 continuous experiments for scenario 1 (benchmarking)

MS5 M24 continuous experiments for scenario 2 (minimal RINA network)

MS6 M26 continuous experiments for scenario 3 (medium size RINA network)

MS7 M28 continuous experiments for scenario 4 (large size RINA network)

Table 3: Milestones for experiment 1-1

Minimum RINA Network: This scenario uses the minimum RINA network (2 hosts, 2 borderrouters, 1 interior router, cf. Figure 13) with an associated strategy for the experiment. The DMSwill be a full configuration as shown in Figure 12.

Medium Size RINA Network: This scenario uses a medium size RINA network, for instance theEuropean network used in experiment 3 as shown in Figure 20 and an associated strategy for theexperiment. The DMS will be a full configuration as shown in Figure 12.

Large Size RINA Network: This scenario uses a large size RINA network, for instance theEuropean network used in experiment 3 as shown in Figure 21 and an associated strategy for theexperiment. The DMS will be a full configuration as shown in Figure 12.

3.2.6 Planning

The milestones for the design and specification of experiment 1-1 are shown in Table 3.

3.3 Experiment 1-2: Deploy new Service in Existing Network

3.3.1 Objectives

This experiment assesses the DMS’s capability to add a new network service to an existing RINAnetwork. The starting point is a RINA network, which can be built using the strategies of experiment1-1. Once the network is built, a new network service (a new DIF) is injected into the existingnetwork. The configuration and scope of this DIF is provided by the operator, e.g. a planning task.This experiment has only 1 step: add a new DIF to an existing network.

The configuration and scope of the new DIF is handled by the DMS. A single strategy shouldbe specified, which takes the operator’s planning specification and, once triggered, creates the DIF.

The experiment is finalised by a second strategy, which deploys a number of tests in the newlycreated DIF to evaluate the correctness of its configuration and scope. Those tests can be active

30

DRAFT




(deploy a test service and evaluate correctness) or passive (analyse the configuration of the DIFagainst the given planning specifications).

The strategy which adds a new DIF can be configured for the three RINA network scenarios.The experiment can then be run for the benchmark scenario and all three RINA network scenarios.We do not anticipate different strategies for the different scenarios.

The initial trigger for adding a new DIF comes from the operator’s OSS/NMS. This will besimulated by a skeleton OSS/NMS trigger application in the DMS. Evaluation of added DIF in thebenchmark scenario can only be performed passively.


This experiment uses the KPIs introduced in section 3.1.5 in terms of speed, touch, complexity (ofthe strategy), and degree of automation.


• the speed a DIF is added to the network (the complete transaction from strategy trigger towhen the DIF is up and running)


• touches and degree of automation, evaluated as the number of touches required add a newDIF


E1-2.1 Speedy DIF add ms or s Service creation and config-uration is often realised byrather complex workflowsthat can take several min-utes or hours (for automatedcreation)

the strategy should run insub-second speed


Currently, CM operationscreate large sets of CMprofiles, one per node

1 operation replicated percreation of IPC process onparticipating nodes


number of touches Creation of new services inthe network is an extremelycomplex and complicatedprocess, with multiple teamsand touches

Ideally zero touch


31

DRAFT











TBD


3.3.3 Software








3.3.4 Testbeds


32

DRAFT






[9]:

Testbed Purpose








3.3.6 Planning


3.4 Experiment 1-3: Deploy Network and Service from Zero

3.4.1 Objectives

This experiment assesses the DMS’s capability to build a new RINA network and add all requirednetwork services, application services, and applications to it. The starting point for this experimentis a topology based on the operator’s topology for the network, the requirements for networkand application services based on operator’s specifications, and a set of applications that must besupported by the network as defined by the operator.

33

DRAFT













At the start, there is no network, i.e. no network node is up and running. The first step is tostart the first network node. Each following step adds a new network node according to the giventopology. Then, the required services are added (e.g. related DIFs created). Finally, all requiredinfrastructure for the applications are deployed in the network. For the applications, we will use thestandard IRATI applications rina-echo-time and rina-tgen.

All configurations are handled by the DMS. A single strategy should be specified that createsthe network, adds all required services, and finally deploys the application infrastructure.

The experiment is finalised by a second strategy, which deploys a number of tests in the newlycreated network to evaluate the correctness of its configuration and scope. Those tests can be active(deploy a test application and evaluate correctness) or passive (analyse the network’s configurationagainst the given operator’s specifications).

The creating strategy can be configured for the three RINA network scenarios. The experimentcan then be run for the benchmark scenario and all three RINA network scenarios. We do notanticipate different strategies for the different scenarios.

The initial trigger comes from the operator’s OSS/NMS. This will be simulated by a skeletonOSS/NMS trigger application in the DMS. Evaluation the network in the benchmark scenario canonly be performed passively.


This experiment uses the KPIs introduced in section 3.1.5 in terms of speed, scale, time and cost,touch, complexity (of the strategy), and degree of automation.


• the speed a node is added to the network (the complete transaction from strategy trigger towhen the node is up and running)

34

DRAFT




• the speed a whole RINA network is built (the complete transaction from strategy trigger tonetwork up and running), very likely determined by the number of similar operations themanagement strategy must realise

• the speed a DIF is added to the network (the complete transaction from strategy trigger towhen the DIF is up and running)

• the scale of the required DMS (one or more executors, possibility of parallelising the actionsfor adding nodes)

• time and cost to scale the DMS up or down (e.g. per RINA network scenario or when runningdifferent scenarios in sequence)


• touches and degree of automation, evaluated as the the number of touches required to build acomplete RINA network and deploy a new DIF

3.4.3 Software








35

DRAFT





E1-3.1 Speedy node add ms or s Node creation and config-uration is often realised byrather complex workflowsthat can take several minutesup to an hour (for automatedcreation)

the strategy should run insub-second speed for singlenode creation

E1-3.2 Speedy networkcreation

ms or s or min Creation and configurationof a whole network can,depending on the networkcomplexity, take severalhours to several weeks

Even the large RINA net-work should be created inminutes, rather than hours

E1-3.3 Speedy DIF add ms or s Service creation and config-uration is often realised byrather complex workflowsthat can take several minutesor an hours (for automatedcreation)

the strategy should run insub-second speed

E1-3.4 DMS scale Number of strategy execu-tors

Multiple workflows executedin parallel (or realised byteams working in parallel)

Ideally only 1 executorrequired (sufficient speed),more (with parallelisedexecution) if speed KPIcannot be met

E1-3.5 DMS Scale Out s and CPU usage Scaling out a managementsystem that is in operationis often impossible or ex-tremely costly (takes hours,requires multiple touches, isprocessing intensive)

Scaling out the DMS shouldbe done in seconds, with 1or zero touch, and minimalCPU costs (for the scalingitself)


Currently, CM operationscreate large sets of CMprofiles, one per node andservice

From early experiments, weestimate 4-5 different oper-ations per node replicatedfor each node to create thenetwork plus deploying therequired network services


number of touches Adding nodes and servicesoften requires multipletouches, realised by multipleteams. Early prototypes forVNF deployment can operateon a minimum (potentiallyzero) basis.

Ideally zero touch


36

DRAFT











TBD


3.4.4 Testbeds



The DMS will be deployed along the IRATI Demonstrator.In summary, this experiment may use the following testbeds from the selection made in D4.2

[9]:

Testbed Purpose







37

DRAFT





3.4.6 Planning











3.5 Experiment 1-4: Add new Network Node - Zero touch

3.5.1 Objectives

This experiment assesses how many touch points are required to add a new node to an existingRINA network. We consider any required manual action or necessary interaction of the DMS withan administrator as a touch point. Ideally, the DMS can add a new node with zero touches, i.e. fullyautomated. It is important for this experiment to establish how close the DMS can be to a zerotouch management system and what information is required to achieve that. Once described, it willalso be of value to minimise the information required.

The starting point for this experiment is a network of at least 1 node, since the first node mightrequire more configuration than any following node. The experiment can then run for every nodedescribed in a given topology. Ideally, the DMS can add all new nodes automatically.

All configurations are handled by the DMS. A single strategy should be specified that addsa new node to the network. After adding a new node, a second strategy should be triggered toevaluate the correct configuration of the newly added node against the given topology.

The initial trigger comes from the operator’s OSS/NMS. This will be simulated by a skeletonOSS/NMS trigger application in the DMS. Evaluation of the network in the benchmark scenariocan only be performed passively.

38

DRAFT





This experiment uses the KPIs introduced in section 3.1.5 in terms of touch and complexity (of thestrategy).


• touches and degree of automation as in the number of touches required to build a completeRINA network and deploy a new DIF




number of touches Beside the (not counted)original touch (installation ofhardware node or trigger forsoftware installation), addinga node often requires multi-ple touches for configuration.Early prototypes for VNFdeployment can operate ona minimum (potentially 0)touch basis.

zero touch


Currently, CM operationscreate large sets of CMprofiles per node

From early experiments, weare estimate 4-5 differentoperations per node


3.5.3 Software





39

DRAFT





IRATI [5] RINA Implementation for Linux O/S GPL and LGPL






TBD





3.5.4 Testbeds




[9]:

Testbed Purpose


40

DRAFT










3.5.6 Planning











3.6 Summary

In this section we have detailed the ARCFIRE experiment 1 focusing on multi-layer coordinationin a converged service provider network. We have defined the main objective, the current problem,and the solution that a multi-layer approach should provide in the context of RINA. We havealso detailed why we are focusing on configuration management. The experiment then is basedon a 4-tuple of aspects, KPIs, scenarios, and use cases. This tuple lead to the definition of 4

41

DRAFT




sub-experiments covering the four described scenarios, providing quantitative results for all fouruse cases, specifying individual KPIs and addressing all described aspects.

42

DRAFT




4 Experiment 2: Deploying resilient, virtualised services over hetero-geneous physical media

4.1 Introduction

Experiment 2 will explore the differences between RINA and TCP/IP when deploying resilient,virtualised services over heterogeneous physical media. The scenario considered here is mobileusers in different parts of Europe connecting over LTE/5G or Wi-Fi access technology. In depthdescriptions of LTE Advanced and WiFi are provided in [1]. Some analytical work has alreadybeen performed on mobility in RINA [10], which focused on comparing RINA to LISP and MobileIP. ARCFIRE experimentation will take a big leap by comparing a RINA 5G solution to the stateof the art in LTE Advanced protocols. Considerations for a 5G deployment based on RINA areprovided in Section 4 of D2.2 [2]. From these considerations, we extract the following requirementsfor an efficient RINA 5G deployment that will be taken as baseline for experiment 2 of ARCFIRE:

• Removal of anchor points: LTE uses either the P-GW or S-GW as anchor points andestablishes tunnels to the UE. Trying to mimic the use of anchor points in RINA wouldincrease complexity and reduce its efficiency. Rather, RINA points to a routing-baseddistributed management solution [11] to be implemented.

• Enhanced capability at the eNodeB: With the removal of the anchor point, the only locationleft that has the opportunity to correctly measure network usage by the end users is directly atthe base station. This means that service mobility as provided by an eNB and an associatedeNodeB Cloud (eNBC) will be mandatory.

4.2 Objectives

RINA’s simple architectural solution to multi-homing and mobility will be evaluated experimentallyagainst the current state-of-the-art of LTE and Wi-Fi networks. The most important aspectsconsidered are reducing overhead between the end user and the base station, the total signallingoverhead in the network, handover duration and packet loss during handovers. We will also evaluatethe impact on the core network, such as routing table sizes.

The key objectives are to find where RINA will perform better or worse than the custom-builtLTE solution. An observation is that the generality of the RINA architecture leads to the overallexpectation that the benefits will increase the further we move up the network hierarchy (i.e awayfrom physical layer constraints) the larger we consider the network, in terms of end user applications,deployed routers, management domains (think Autonomous Systems), and so on.

From this general observation, areas where we expect RINA to outperform 5G proposals builton SDN solutions are a general reduction of complexity, reducing the number of elements necessaryto operate the network to the same level of efficiency and performance. Replacing tunnels by a

43

DRAFT




routed solution removes a complex element (the MME) from the equation. A key objective will beto quantify this by measuring network characteristics across the board.

Another area where RINA should perform very well is reliability. Using a routed solution insmall DIFs, we can tailor routing updates to react very fast to changes in network connectivity.Whatevercast naming will also simplify managing handovers and mitigate connection interruptionsthat cannot be handled within a DIF (when the graph becomes separated into different components).Combined with the application addressing, we expect measurable decreases in connection downtimein the presence of averse network conditions. This work has already been investigated in thePRISTINE project at a smaller scale [12]. With the improved software frameworks that aredeveloped by ARCFIRE, we will be able to provide results in much more detailed scenarios.

Sticking points for a RINA deployment will be where hardware constraints will be felt themost, close to the transmission equipment. LTE has a very efficient solution for reducing overheadbetween the eNodeB and the UE in the RObust Header Compression (ROHC) technology used byPDCP, replacing L3 and L4 headers (IP/UDP/RTP) by a very small token, reducing 40 (or even60 bytes in case of IPv6) to 2-3 bytes [13]. RINA’s layered structure, with encapsulation at eachlayer to maintain transparency may introduce additional overhead at this point. As part of thisexperiment, we will investigate policies for header compression to reduce the Protocol ControlInformation in the PDUs on the eNodeB-UE link.

4.3 Experiment scenario

Experiment 2 considers mobile end users accessing a variety of services. There will be 4 servicesavailable in the experiments (Web, Voice, Gaming and File Transfer). These users will reside acrossEurope, interconnected over a single core network. A scenario involving multiple core networks isinvestigated in experiment 3. For the LTE based experiments, we will use the nginx web-server, theMidori web-browser, VLC for the video service, ioquake for the gaming service and Filezilla forthe file transfer. For the RINA experiments, ports are already available for nginx (the port will bereleased as open-source in the next month) and ioquake3 [14]. The other services will be validatedby comparing statistics gathered with the RINA tools like rina-tgen and rinaperf (described in [15],Section 3.1), which can be run by means of the Rumba framework. If additional RINA ports shouldbecome available (e.g. a browser of a video streaming server/client), they will be adopted in theexperiments as well.

In the mobile edge, we will investigate three access technologies: Fixed (Gb Ethernet), LTEand Wi-Fi and consider handovers within the same access technology and cross-technology andwithin an between network operators. This includes handovers between base stations of the sameoperator (X2 and S11 interface) and between different operators.

In the metro and core network, we will build on 1G and 10G Ethernet links for some bare-metalmeasurements. For scaled up experiments, we will use virtual machines in GENI and iLab.t(interconnected wherever possible) to perform experiments that are larger in terms of network hosts,but less demanding in terms of bandwidth usage. Such experiments will run applications that can

44

DRAFT




work also with low bandwidth, like web-browsing and measurement tools (ping, netperf, rina-tgen,rinaperf, ...).

Experimentation with routing policies tailored for mobility (that produce routing updates morefrequently), will explore good trade-offs for the maximum size of a DIF given a certain routingpolicy and a certain mobility rate (maximum size will be limited by the time to disseminate therouting updates when a handover happens). The influence of having a routing policy in every DIFon the routing table sizes will also be investigated.

Figure 14: Baseline LTE experiment

The first experiment will be a baseline measurement of the LTE user and data plane to be usedfor reference values. OpenAIRInterface (OAI) [16] is the software of choice as it is open source,easily available and has an active development team behind it. This experiment will also serve as anintegration test validating the OpenAIRInterface deployment. The deployment is shown in Figure14. It contains the following elements:

• The User Equipment (UE). The UE will run the OAI soft UE client, two end-user applications(VLC client and netperf).

• eNodeB’s (2). These two servers will run the OAI soft eNodeB module and the UE willperiodically move between the two base stations (simulated through disconnects over thephysical link).The two base stations are not directly connected since the X2 interface is notimplemented yet by OAI. Should such an interface become available during the project, thisexperiment might be rerun to include the results.

• the S-GW and P-GW. This will use the OAI soft EPC implementation.

• the MME, also from the OAI soft EPC implementation.

Each element will be installed on a bare metal server to provide accurate traffic measurementsin abscense of interference of schedulers associated with running elements in virtual machines. We

45

DRAFT




will capture the entire experiment’s traffic at all interfaces and analyse the traces to measure KPIsrelated to handovers and signaling overhead. A small sister experiment with the same topologywill be conducted on the wireless testbed to validate the results on a real radio link. The reasonfor this is that the wireless testbed is not always available (due to physical interference betweenexperiments, the testbed is usually entirely reserved for a single experiment at a time).

Figure 15: End-to-end LTE experiment

After this experiment, the deployment will be mirrored and run in both an LTE and RINAdeployment. The combined network is shown in Figure 15. Note that the eNodeB’s are directlyconnected, because this will have benefits for the RINA deployment. The eNodeB Clouds (eNBCs)will run on the same node and user data can be moved between these two nodes when the usermoves. Usage statistics will probably be monitord at the shim IPCPs on the eNodeBs. The MMEnodes will not be needed in the RINA setup.

The remainder of experiment2 will focus on scaling up the RINA scenario to 4 connected accessnetworks (each having 1 or 2 EPCs) with some 30 eNodeBCs between them, serving a combinedtotal of 1000 UE’s. The single metropop will be replaced by a 3 meshed MANs of up to 8 nodeseach interconnected by a core network of up to 5 nodes interconnected by (preferably) 10G links.The total final experiment will therefore use the following estimaed number of nodes in the testbed:5 core nodes, 20 metro nodes, 30 eNodeBC nodes and some 10 UE nodes simulating 1000 UE’s.

4.4 Metrics and KPIs

The objectives of this experiment are to quantify the following

• Handover interruption time: Handover interruptions can be derived from capturing data atthe application (for instance netperf or rina-tgen), and the physical interfaces of the deviceunder test. We will use tcpdump/pcap to capture the traffic traces.

• Packet loss during handovers: Packet loss can be derived in a similar way as the handoverinterruption time.

• PCI overhead: Protocol analysers such as Wireshark and tcpdump will allow us to separatecontrol packets in the IP over LTE scenario. In the RINA scenario, we will log statistics fromthe IPC processes themselves for analysis.

46

DRAFT




• Routing scalability: Routing tables can be dumped to storage at certain intervals and all thechanges can be logged. These functionalities are already supported by the available RINAimplementations.

• Messaging overhead: We will use capture data from the physical interface to investigate themessaging overhead that a handover causes. Again we can use tcpdump/pcap to capture andanalyse the traffic traces.

• Average bitrate: Applications such as netperf or rina-tgen can be used to measure the averagebit rate to complete a file transfer.

These KPIs are summarised in Table 13.


Handover interruption time ms 10-20 ms [17] ∼ 20 ms measured in the testbed

Packet loss during handovers number of SDUs 0 0

Routing scalability RIB entries IPv4 / IPv6 RIB/FIB Guarantee linear or sub linear scaling

Messaging overhead during handovers pps, b/s LTE CP traffic reduce control traffic

PCI overhead bits ROHC comparable to ROHC

Average bit rate to complete a filetransfer

Mb/s technology-dependent

Table 13: KPIs for experiment 2

4.5 Software

The software components that will be required for experiment 2 are listed in Table 14 for itsexecution. These include three available RINA implementations, the ARCFIRE measurement andexperimentation framework (Rumba, documented in [15]), various testbed-specific tools and allthe software necessary for the baseline LTE-based experiment. The possible use of different RINAimplementations is expected to carry an added value to the experiments, as different design decisionmay result in different performance and/or scalability properties.

4.6 Testbeds

The experiment will use the testbeds in Table 15 from the selection made in D4.2 [9]. Thesetestbeds have been chosen taking into account previous know-how of T4.3 partners, and to supportthe number of nodes required during the various phases of the experiment (i.e. ranging from ∼30to ∼1000 nodes). Moreover, running the experiment on both a VM-based testbed (GENI) and abaremetal-based testbed (Virtual wall) has been considered an important factor for the evaluation of

47

DRAFT





IRATI [5] Open source RINA Implementation for OS/Linux, inherited by FP7-IRATI andFP7-PRISTINE

GPLv2, LGPL

rlite [18] A lightweight Free and Open Source RINA implementation for GNU/Linuxoperating systems, focusing on stability, ease of use, and performance.

GPLv2, LGPLv2.1

Ouroboros POSIX compliant RINA implementation GPLv2, LPGLv2.1

jFed [19] Java-based framework for testbed federation MIT

Rumba [8] jFed compatible experimentation framework LGPLv2.1

OpenAIRInterface [16] Free Open Source Software for EPC, access network and user equipment of3GPP cellular networks

GPLv3

Quagga [20] Free Open Source routing platform GPLv2 (or later)

Table 14: Software for experiment 2

RINA software stability, as the sensitive difference in I/O timings 1 can result into a larger softwaretesting coverage.

Testbed Purpose

iLab.t Virtual wall Host the first experiments up to large scale testing. Measurements of bandwidth, delay, jitter andburst characteristics on dedicated links and bare metal hardware

w-iLab.t Validation of the LTE setup and measurements of eNodeB-UE bandwidth usage

GENI / planetlab Europe Emulation of a continent-wide core network, evaluation of scalability (routing table sizes) on VMs

Table 15: Testbeds for experiment 2

4.7 Planning

Table 16 details the expected milestones for experiment 2 throughout the execution of ARCFIRE.The first period (M16-M19) will be dedicated to the setup and KPI measurement for an LTE basednetwork, in order to collect statistics that can be used for later comparison with RINA experiments.From M19 on, experiments will make use of RINA, starting will small-scale setups (∼ 30 nodes) ina single testbed and scaling it up to ∼ 1000 nodes and using multiple testbeds.

The risk factors for experiment 2 are mostly related to the maturity of the experimentationsoftware. OpenAIRInterface and the used RINA prototypes do not (and cannot) have the samedegree of maturity as well-established IP-based and LTE software. This relative lack of maturitycan reveal itself in two ways. Moving an implementation from its development environment to thetestbeds may reveal corner cases (or in general code paths never taken before) that are related tothe specific hardware it is running on. Machines with different number of processors, processor

1I/O operations involving Virtual Machine can be an order of magnitude slower in terms of throughput and/or latencywhen compared to I/O on baremetal applications

48

DRAFT





Deploy OpenAIRInterface in iLab.t M16 Installed and verified the setup of an LTE-Advanced EPCand E-UTRAN with a single UE, 2 eNodeB’s, one MME,one S-GW, one P-GW and a core node hosting a singleservice

Connect two EPC deployments M17 The above deployment will be mirrored and connectedover a single core node

Document baseline measurements M19 This will give baseline measurements for the LTE sce-nario for the identified KPIs

Deploy RINA software in LTE scenario and get baselinemeasurements

M19 This will conclude the first small scale RINA experi-ment.

Scale up RINA deployment to mid-scale experiment M22 4 connected access networks, ∼ 20 UE’s, ∼ 10 eN-odeB(C)s, 2 services

Document measurements from the mid-scale experi-ments

M23 This will conclude the mid-scale experiments

Scale up RINA deployment to large-scale experiment M24 4 connected access networks, ∼ 1000 UE’s, ∼ 30 eN-odeBC, ∼ 10 services

Design multi-testbed deployment M25 This will depend on the results from the previous experi-ments

Execute multi-testbed deployment M27 This will depend on the results from the previous experi-ments

Wrap up experimentation M29 Detailed analysis of the results and preparation for publi-cation of the final results.

Table 16: Planning of experiment 2

49

DRAFT




microarchitecture, memory speed, caches size and NIC models can reveal timing issues and dataraces that were previously less visible. Scaling up the implementation will increase the rate atwhich any deficiency (memory leak, dereferencing a pointer just outside an allocated range) revealsitself. In addition to that, the likelihood of software failures increases (or at least does not decrease)as the duration of experiment increases.

As a quantitative indication of the largest and longest RINA experiments conducted to dateon the available implementations, IRATI has been scaled up on networks of Virtual Machinescontaining about 40 nodes; rlite has been deployed on networks of VMs with about 90 nodes.Moreover, these test networks have been operational only for some hours, while ARCFIRE targetsat least a week of uninterrupted operation.

These risks are mitigated in two ways:

• T3.4 (which runs until the end of the project) is dedicated to fixing bugs, memory leaks andother problems found during the various planned phases of the experiment.

• Three independent RINA implementations are available to be used for experiment 2, sothat fallback options are available if major issues should arise with either one of theseimplementations.

50

DRAFT




5 Experiment 3: End to end service provisioning across multiple net-work providers

5.1 Objectives

The main objective of this experiment is to assess the the RINA IPC API in terms of end-to-endservice provisioning complexity as compared to state-of-the-art network technologies currentlyin use in converged operator networks. This experiment will focus on the operation of severalservice DIFs supported by the core network of one or more service providers; these service DIFsprovide QoS guarantees to the applications using them, and request suitable QoS assurances ontheir supporting layers, the backbone DIF. Also, in order to support the guaranteed QoS levels, eachDIF will require specific sets of policies and parameters. Different scenarios will be considered,each one targeting different aspects of the experiment, and thus requiring different sets of policies,parameters, etc. These scenarios and the features to be taken into account are based on the analysiscarried out in deliverable D2.2 [2], sections 3.7 and 5.3. As discussed there, the focus will be onthe allocation of flows between applications or services by using their names, defining a contractfor an acceptable level of service for each flow in terms of technology-agnostic parameters; thismechanism will be shown to be of use between DIFs, allowing the communication of requirementsto the supporting DIFs and freeing the layers from the need to disclose internal technological detailsto the other layers. Allocation of flows between multiple operator networks with end-to-end qualityof service guarantees becomes possible and will be explored.

The second goal of Experiment 3 is to evaluate network renumbering. Multiple networkconfigurations will be analysed in which renumbering takes place in one or multiple DIFs at once.Dynamic renumbering has applications for mobility (moving hosts get new addresses that reflecttheir current location in the different DIFs they are members of) and also for modifying a DIFaddressing plan in case the network operator wished to do so.

Last but not least, Experiment 3 will analyse how RINA applies to an Open Mobile EdgeComputing (OMEC) use case. In OMEC use cases some applications may be hosted in the serviceprovider data centres distributed through the mobile access network. These applications are onlyavailable to certain uses via secure slices. This scenario will focus on the analysis of the DIFallocator to locate and application processes and create new DIFs, secure access to DIFs and RINAsupport for distributed mobility management to realise OMEC use cases.

In the following sections theses scenarios are described in detail.

5.2 Quality of Service levels guaranteed by service provider network

5.2.1 Objectives

Figure 16 depicts the core network of a service provider supporting multipurpose service DIFs. Inorder to design the best suited policies for this scenario, the distance between nodes and traffic

51

DRAFT




characteristics are the basic parameters. Two network sizes will be considered, a smaller one with11 backbone nodes and a larger one, with up to 40 nodes. For the latter, two service providerswill be considered, with the abstract layering structure shown in figure 17. The distance betweenthe nodes will be considered as added delay between the point-to-point links. Regarding thetraffic characteristics, each DIF will have to transport different types of traffic, with different QoSrequirements.

Figure 16: Guaranteed QoS levels experiment: abstract layering structure, showing the differentDIFs to be considered

The QoS guarantees would be achieved through a set of policies for the RMT and the EFCPimplementing the QTAMux, as described in D2.2 [2]. It provides quality differentiation andscheduling for different traffic classes.

Figure 17: Guaranteed QoS levels experiment: abstract layering structure, showing the differentDIFs to be considered. Provider edge routers are shown

The DIFs shown in figure 16 will have to deal with different traffic profiles, and thus the policies

52

DRAFT




for the RMT and EFCP would need to be adapted:

• Backbone core DIF will transport highly aggregated traffic. Thus, for this scenario only oneQoS level would be needed.

• The service provider DIF on top will transport several types of traffic. In this DIF is where theQTAMux enters the picture, providing the scheduling and delay/loss multiplexing requiredto satisfy the level of service requested by applications.

• Several VPN DIFs for different customers will be supported by the service provider DIF. Inthis case, the DIF will support the same QoS levels that the service provider DIF will offer,but no QTAMux will be required, as it will be the supporting DIF which will be doing that.


The KPIs for experiment 3 are given in Table 17.


Delays Delays measured per QoS class Hard to guarantee differential delaybetween QoS classes at high loads[21]

Statistical bounds on delay per QoSclass, even when offered load isequal to 100%

Jitter Jitter measured per QoS class andflow

Hard to guarantee differential jitterbetween QoS classes at high loads[21]

Statistical bounds on jitter per QoSclass, even when offered load isequal to 100%

Losses Losses measured per QoS class Hard to guarantee differential packetloss between QoS classes at highloads [21]

Statistical bounds on packet loss perQoS class, even when offered load isequal to 100%

Table 17: KPIs for guaranteed QoS levels experiment

5.2.3 Software

This experiment will require the following software items for its execution, summarised in Table18.

5.2.4 Testbeds

The performance-sensitive nature of this experiment requires the use of bare metal hardwareresources if possible. Only a moderate number of resources is required to carry out the tests,without the need of supporting wireless interfaces. Hence, the Virtual Wall is the best testbed optionfrom the selection made in D4.2 [9]:

53

DRAFT




Software package Description License


rina-echo-time [5] Measure jitter, delay and packet loss GPL




Table 18: Software required for guaranteed QoS levels experiments

Testbed Purpose

Virtual Wall Provides access to enough bare metal servers to carry out the experiments described in this scenario

5.2.5 Experiment scenario

Two scenarios have been designed in order to gather data for the effect of network size in the KPIsdescribed above. Figure 18 shows the backbone connectivity of the small scale setup. For the largersetup, two similar backbone connectivities will be used for both network providers.

Figure 18: Guaranteed QoS levels experiment: small DIF experiment, backbone DIF connectivity

The most relevant characteristics of the policies for the backbone DIFs are summarised in thefollowing paragraphs.

54

DRAFT




• Data transfer: No retransmission control policies will be considered.

• Support for multiple QoS classes: In these DIFs only one QoS class will be offered.

• Multiplexing and scheduling policies: Since the N-1 DIFs will be point to point DIFs, andonly one QoS class will be offered, a simple FIFO scheduling schema will be used.

The characteristics of the policies for the service provider DIFs are summarised in the followingparagraphs.

• Data transfer: No retransmission control policies will be considered. Sliding window flowcontrol, with ECN based congestion control will be used.

• Support for multiple QoS classes: In these DIFs, traffic from different applications or DIFs,with different QoS requirements will make necessary to offer several QoS classes. For thescenarios considered here, it will be necessary to offer the same classes as the layer on top,the VPN DIF or app-specific DIF or public internet DIF.

• Multiplexing and scheduling policies: For these DIFs it will be necessary to use theQTAMux policies, given the multiple QoS classes to be supported.

Finally, for the the VPN DIFs or app-specific DIFs or public internet DIFs the following policieswill be considered:

• Data transfer: In this DIF, taking into account the possibility of packet losses due to theQTAMux scheduling in the DIF below, retransmission control policies will be made availablefor the applications that require it. Since burstiness is to be expected, sliding window flowcontrol will be used, associated with ECN based congestion control.

• Support for multiple QoS classes: In this DIF, traffic from different applications withdifferent QoS requirements, will make necessary to offer several QoS classes.

• Multiplexing and scheduling policies: For this DIF no scheduling policies will be necessary.The DIF below will take care of the scheduling.

Table 19 shows the different types of traffic that will be considered in this scenario. Thesetraffic profiles will be translated into configuration parameters for the QTAMux.

Thus, three applications would be used to gather the results: real-time voice, video on demand,file transfer; each one is a modified version of the rina-echo-time application.

Figure 19 shows the layering for a scenario with two network service providers. The VPN DIFwill support the traffic profiles defined above matching them with its QoS classes. The Edge ServiceRouters aggregate traffic from different CPEs; therefore, each one will support several flows foreach type of application. These flows will be aggregated and transported by the backbone DIF.

55

DRAFT




Urgency Type of application Flows allocated (%) Cherish level

Urgent traffic Real-time voice 20% Medium

Medium urgency traffic Video on demand, web brows-ing

40% High

Low urgency traffic File transfer 40% Low

Table 19: KPIs for guaranteed QoS levels experiments

Figure 19: Guaranteed QoS levels experiment: layering of the scenario

56

DRAFT





MS 1 M16 All software for experiments works on demonstrator, initial results using small-scale and large-scaledeployments

MS 2 M18 Experiments carried out at VWall testbed using small-scale deployment

MS 3 M22 Experiments carried out at VWall testbed using large-scale deployment for one network service provi-der

MS 4 M28 Experiments carried out at VWall testbed using large-scale deployment for two network serviceproviders

Table 20: Milestones for guaranteed QoS levels experiment

To carry out the experiments, many flows for each type of application will be allocated betweenall possible node pairs (in the case of the small scenario), and between nodes that are not tooclose in the case of the larger one. The delays, jitter, and packet losses will be measured by theapplications.

5.2.6 Planning

We foresee the milestones reported in Table 20 in the development of the guaranteed QoS levelsexperiment.

5.3 Dynamic and seamless DIF renumbering

5.3.1 Objectives

One of the requirements for achieving seamless mobility and at the same time keep routing efficientand forwarding tables small in DIFs is the ability to renumber IPC Processes as they move throughthe network. An effective and scalable addressing strategy assigns addresses to IPCPs that arelocation dependent (that is, the address reflects the position of the IPCP with respect to an abstractionof the DIF connectivity graph). As the IPCP moves through the DIF, it will reach a point whereits current address no longer reflects his position accurately (hence it is no longer aggregatable).The IPCP needs to get a new address assigned in a dynamic fashion, following a procedure thatguarantees that the flows supported by the IPCP are not impacted.

The procedure to change the address of a network entity is usually known as renumbering, andis not a seamless one in IP networks: IP addresses need to be assigned to interfaces on switches androuters, routing information must be propagated, ingress and egress filters must be updated - aswell as firewalls and access control lists, hosts must get new addresses and DNS entries have to beupdated.

An overview of the problems associated to renumbering of IP networks is provided in [22].Since TCP and UDP connections are tightly bound to a pair of IP addresses, changing any of themwill destroy the flow. Since DNS is an external directory - not part of the network layers - the

57

DRAFT




renumbering process usually leads to stale DNS entries pointing to deprecated addresses. Evenworse, applications may operate through the direct use of IP addresses, which will require anupdate in the application code, its configuration or both. Router renumbering usually requires anexhaustive manual and error-prone procedure for updating control plane Access Control Lists orfirewall rules. Moreover, static IP addresses are usually embedded in numerous configuration filesand network management databases [23].

Dynamic renumbering in RINA networks is possible because none of the previous issues canhappen. Flows provided by RINA DIFs are between application process names; since IPC Processesnever expose addresses outside of the DIF renumbering events are not externally visible. All DIFshave internal directories that match names of the applications registered to that DIF to the addressesof the IPC Processes they are currently attached to; therefore renumbering requires the update ofthis mapping, which is just a normal operating procedure in the DIF (no special function required).Access control rules are also setup according to application names, and therefore need not to beupdated when IPCPs change addresses. Last but not least, Network Manager applications interactwith Management Agents in the managed systems via its application name, and hence are notexposed to IPCP address changes.

The main goal of this experiment is to understand the behaviour of dynamic renumbering onRINA networks, analysing what are the limitations and tradeoffs to make renumbering events in aDIF invisible to the applications using it. This experiment will explore dynamic renumbering inseveral configurations of DIFs extracted from D2.2 [2], using this technique in two main use cases:

• Hosts moving through a provider’s access network. Addresses in the IPCPs of the mobilehost will need to be updated as it changes its point of attachment. In this scenario onlya subset of the addresses of IPCPs in one or more DIFs need to be updated, but that mayhappen rather frequently depending on the speed of the mobile hosts.

• A provider updates the addressing plan of one or more DIF(s) in its network. This mayhappen because the current addressing plan no longer scales, or because the provider wantsto change to an addressing plan that is more efficient given the network connectivity graph.In this scenario all the addresses of IPCPs in one or more DIFs need to be updated, but veryinfrequently.


The objectives of the experiment are to quantify i) the degradation in the level of service perceivedby applications using flows when renumbering takes place in a DIF and ii) the overhead andperformance of the renumbering procedure. As later explained in section 4.3.5, experiments willtake into account realistic scenarios but also more extreme scenarios to understand the limitationsof dynamic renumbering (e.g. in which a network is continuously being renumbered). That is whythe seamless renumbering KPI targets in table 21 refer to the realistic use cases.

58

DRAFT





Seamless renumbering: Latency Increased latency while thenetwork is being renum-bered

Application flows breakwhen renumbering

Less than 5% in realisticuse cases

Seamless renumbering: Goodput Decreased goodput whilethe network is beingrenumbered


Less than 5% in realisticuse cases

Seamless renumbering: packet loss Packet loss due to renum-bering events


Zero in realistic use cases

Renumbering overhead Average extra entries inIPCP forwarding tablesdue to renumbering events,as a function of renum-bering period and DIFsize

Renumbering cannotbe fully automated andseamless

Understanding the trade-offs between renumberingperiod, DIF size and for-warding table size

Renumbering speed Time to complete a renum-bering event (until oldaddress is deprecated)

Renumbering cannotbe fully automated andseamless

Understanding the perfor-mance of the renumberingprocedure, and how it mayimpact seamless renumber-ing metrics

Table 21: KPIs for renumbering experiments







Table 22: Software required for renumbering experiments

5.3.3 Software


• IRATI RINA implementation. It will be used to implement multiple RINA-enabled systemsand run a variety of DIFs according to the experiment configuration depicted in section 4.3.5.Policies already present in the IRATI repository will be used to carry out standard DIFprocedures. A few new namespace management policies will be developed to trigger therenumbering events according to different strategies (periodically, whole DIF at once, onlymobile hosts).

• rina-echo-time. The rina-echo-time application will be used to measure the latency andpacket loss perceived by a flow user while renumbering takes place.

59

DRAFT




• rina-tgen. The rina-tgen application will be used to measure the goodput perceived by a flowuser while renumbering takes place.

• Demonstrator. The demonstrator will be used to test the correct behaviour of the softwarebefore deploying it on FED4FIRE+ testbeds. The demonstrator allows the experimenter tocarry out multiple experiment runs in a controlled environment with enough scale (up to 100RINA-enabled Virtual Machines).


5.3.4 Testbeds

We do not foresee the use of wireless resources to carry out the renumbering experiment, thereforethe Virtual Wall will be its main testbed of reference as reported in D4.2 [9]. The Virtual Wallcan provide access to bare metal hardware to run the RINA implementation and the applicationsrequired for the experiment, providing access to an adequate number of resources.

Initially the jFed experimenter Graphical User Interface (GUI) will be used to reserve and setupthe resources in the FED4FIRE+ testbeds. Once it becomes available, the experimentation andmeasurement framework under development in WP3 will provide a more automated front-end forjFed as well as other IRATI deployment and configuration tools such as the Demonstrator.

In summary, this experiment may use the testbeds in Table 5.3.4.

Testbed Purpose

Virtual Wall Provides access to enough bare metal servers to carry out renumbering experiments

Table 23: Testbeds for renumbering experiments


Renumbering experiments will be based on two scenarios: single DIF scenarios, designed totest how renumbering behaves within a single DIF; and multi-DIF scenarios, designed to testrenumbering across a whole network (set of DIFs). We foresee two versions of the single DIFscenario: a small one and a large one. The first one, depicted in Figure 20, is modelled after theGEANT network backbone and has 32 IPC Processes. The second one, depicted in Figure 21, ismodelled after the AT&T network layer 3 backbone and has 80 nodes.

Multi-DIF experiments will leverage the ideal RINA converged service provider designsdescribed in D2.2 [2]. In particular we will setup small and large multi-DIF scenarios for the twouse cases considered in the experiment: mobility and change of addressing plan. Figure 22 shows

60

DRAFT




TALOslo STO

COP

DUB

LISMAD

Bern

Rome

SOF

VIE BUC

ATH

VIL

NIC

WAR

PRA

LUX

Riga

BER

LONAMS

BRU

PAR

LJU ZAG

BUD

ANK

VAL

MOSOslo

DUB

LIS

NIC

BRU

ANK

VAL

Riga

Figure 20: Renumbering experiment: small single DIF scenario, layering structure (up), and DIFconnectivity (down)

61

DRAFT




DAL

OKH

ORL

SJUMIA

FW

ALB

COL

PHO

DEN

COL

ATL

LA

LVG

SDG

SLC

AUS

SAN HOU

NOR

GAR

HON

SJO

SFR

SCR

DC

NYCDET

CAM

CHIOMH

KAN

SEA

ANC

POR

SLO

NORANH

RED

OAK

TMPFLA

PMJAK

SPO

MAD

MIL

DAV

DMO

NASHTUL

LRO

GLE

GRA

SPA

MIN

MEM

IND

LOU

BIR

CIN

CHA

RAL

PHIPIT

BAL

NEW

BOH

SYR

BRI

CHE

ALB

WOR

PORMAN

FAR

PRO

Figure 21: Renumbering experiment: large single DIF scenario, DIF connectivity

the DIF structure for the change of addressing plan scenario. The experiment will focus on theresidential customer DIF and below. Several tests will be carried on two versions of the scenario.The first one is illustrated by Figure 23, featuring 41 systems. The second one is depicted in Figure24, and uses 112 systems in a larger scale configuration. The scenario where renumbering supportsmobile hosts will be tested using a similar configuration, in which mobile hosts will be used insteadof CPE routers. Since we are only interested in analysing the change of address events, there isno need to have real mobility using wireless technologies in the experiment setup; mobility eventswill be simplified down to renumbering events occurring in a pattern equivalent to that of multiplemobile hosts roaming through the network.

A namespace management policy will be used to trigger different network renumbering situ-ations. With this policy every IPCP in the DIF periodically changes its address (walking a poolof addresses assigned reserved for each IPC Process). The period is a random variable uniformlydistributed between a minimum and a maximum values (configurable when specifying the policy).Playing with the configuration of the policy it is possible to experiment with DIFs where all itsmembers continuously renumber at different times, or DIFs where some of its members changeits address, or DIFs where all its members change addresses at the same time just once. Figure 25shows the different scenarios and configurations that will be taken into account for the renumberingexperiment.

62

DRAFT




Access Router

PtP DIF

CPE

Edge Service Router

Metro Edge

Router

Metro Edge

Router

Metro BB DIF

Metro service DIF

PtP DIF PtP DIF

PtP DIF PtP DIF

Metro P router

PTP DIF

Residential customer service DIF

Host PtP DIF

Public Internet or App-specific or VPN DIF

Backbone Router

Backbone router

PtP DIF PtP DIF

Backbone DIF

Provider Edge

Router

Provider Edge

Router

PtP DIF


Access Aggregation Service Edge Core Internet Edge

Public Internet or App-specific or VPN DIF

Home DIF

Figure 22: Renumbering experiment: DIF structure for the multi-DIF scenario

Customer Premises Equipment

Access Router

MAN Access Router

MAN Core Router

Edge Services Router

Backbone router

Figure 23: Renumbering experiment: systems for small scale multi-DIF scenario

63

DRAFT





Access Router

MAN Access Router

MAN Core Router

Edge Services Router

Backbone router

Figure 24: Renumbering experiment: systems for large scale multi-DIF scenario

Continuous renumbering

Mobile host moves through the DIF

Renumbering experiments

Change of addressing plan

Single DIF

Multiple DIFs

Small scale

Large scale

Small scale

Large scale

Single DIF

Multiple DIFs

Small scale

Large scale

Small scale

Large scale

Small scale

Large scale Multiple DIFs

Figure 25: Categorisation of scenarios and configurations for the renumbering experiments

64

DRAFT





MS 1 M16 All software for experiments with continuous renumbering scenario works on demonstrator, initialresults using small-scale and large-scale deployments

MS 2 M18 Continuous renumbering experiments carried out at VWall testbed using small-scale deployment

MS 3 M22 Change of addressing plan and mobile hosts scenario experiments carried out at VWall testbed usingsmall-scale deployment (maps to ARCFIRE project milestone MS8)

MS 4 M26 Continuous renumbering experiments carried out at VWall testbed using large-scale deployment

MS 5 M28 Change of addressing plan and mobile hosts scenario experiments carried out at VWall testbed usinglarge-scale deployment (maps to ARCFIRE project milestone MS9)

Table 24: Milestones for renumbering experiments

5.3.6 Planning

We foresee the milestones reported in Table 24 in the development of the renumbering experiments.

5.4 Application discovery, mobility and layer security in support of OMEC

5.4.1 Objectives

OMEC can be defined as “An open cloud platform that uses some end-user clients and located atthe ‘mobile edge” to carry out a substantial amount of storage (rather than stored primarily in clouddata centres) and computation (including edge analytics, rather than relying on cloud data centres)in real time, communication (rather than routed over backbone networks), and control, policy andmanagement (rather than controlled primarily by network gateways such as those in the LTE core).”[24]

D2.2 [2] introduced the Open Mobile Edge Computing Scenario as one of the 5G scenarioswhere RINA could deliver significant value. This experiment analyses the buit-in capabilities inRINA that facilitate the realisation of OMEC use cases. In particular:

• Automatic location of applications regardless of the DIF(s) they are available through.In mobile edge computing scenarios storage, compute, and networking of edge servicesare provided at the edge of the mobile network. In a RINA network, compute and storageresources can be placed wherever needed. RINA can then discover distributed applications(edge services), locate processes and allocate flows to (between) them independent of theirnetwork location.

• Slicing: security and performance isolation. Network slices are isolated with guaranteedsecurity and performance, while sharing the same underlying infrastructure. All of that isoptimised for the delivery of a set of applications (or 5G verticals). RINA already has anative support for scope, slicing, and virtualisation as discussed in sub-section 3.8.1 of D2.2[2]. In essence, the concept of a DIF provides for securable layers (any number of them)

65

DRAFT




whose policies can be tailored to the needs of each tenant and/or application (e.g. virtualisednetwork function).

• Distributed mobility management. Distributed mobility management avoids centralisedmobility anchors providing efficient routing and traffic management (including handovers).Furthermore, it can be used to remove tunnels if possible. RINA already supports (natively)multi-homing and mobility, including multiple possible handover scenarios. Other RINAcore features (scoped routing, topological address and routing, and seamless renumbering)allow a RINA network to efficiently support any type of mobility and distributed mobilitymanagement. Experiment 2 will analyse this topic in depth, Experiment 3 only investigateshow distributed mobility management is used as an enabler of OMEC scenarios.


The objectives of this experiment are to evaluate the behaviour of several built-in RINA featuresfacilitating OMEC use cases, by verifying their correct operation and measuring the performanceof its current implementation. In particular we are interested in the DIF Allocator work to locatea destination application and configure DIFs to access it if needed; as well as in the impact ofhandovers in terms of performance (packet loss, delay, goodput) in a RINA over WiFi scenario(Table 25).


DIF Allocator performance Extra delay incurred inflow allocation due toapplication discovery andjoining relevant DIF

This capability is notsupported by the currentInternet protocol suite

Understanding the perfor-mance of the DIF Allocatorunder load, identifyingtrade-offs in its design

Impact of handover on packet loss Increased packet loss dueto handover (WiFi AP toWiFi AP)

To be measured on testbed Equivalent or less than inthe IP case (understandthe tradeoffs of RINA overWiFi)

Impact of handover on delay Increased delay due tohandover (WiFi AP to WiFiAP)

To be measured on testbed Equivalent or less than inthe IP case (understandthe tradeoffs of RINA overWiFi)

Impact of handover on goodput Loss of application good-put (Mbps) due to handovereffects

To be measured on testbedfor TCP and UDP

Equivalent or less than inthe IP case (understandthe tradeoffs of RINA overWiFi)

Table 25: KPIs for OMEC experiments

5.4.3 Software


66

DRAFT






DIF Allocator [5] Locates applications over a variety of DIFs and collaborates with NMS(s) tocreate new DIFs

TBD




Rumba [8] Tool to setup RINA implementation experiments on FED4FIRE+ testbeds LGP

Table 26: Software required for OMEC experiments

5.4.4 Testbeds

The w-iLab.t wireless testbed provides an ideal environment for the wireless segment of theexperimental scenario depicted in Figure 26. Such a facility provides access to a large number ofWiFi access points - which will be used to model the access routers in the experiment - as well asaround 20 hosts mounted on robots that can move through the facility - used to model the mobilehosts. The Virtual Wall will be used to provide resources for all the fixed routers (core routers, DCgateway, Top of Rack, ISP routers, provider border router) as well as the servers in the experiment.There is connectivity between the w-iLab.t and the Virtual Wall, therefore it is possible to carry outexperiments that use resources from both testbeds.

Table 5.4.4 summarises the testbeds that will be used by the OMEC experiment.

Testbed Purpose

w-iLab.t Hardware resources for the wireless access routers and the mobile hosts, connectivity to Virtual Wall

Virtual Wall Hardware resources for the fixed routers (core, border, ISP, DC) and servers

Table 27: Testbeds for OMEC experiments


The experiment scenario features a provider hosting applications belonging to two different en-terprises in its own datacentre. Several mobile hosts (UEs) belonging to employees of each oneof the two companies are serviced by the provider. UEs connect to applications that are eitheravailable through the public Internet or through the provider’s edge datacentre. UEs are completelyunaware of the location of such services, which are dynamically discovered by the RINA network.Employees can move and attach to different provider access routers without service disruption.Figure 26 shows the configuration of the physical systems in the experiment scenario. The providernetwork consists in 6 wireless access routers, interconnected between them via 2 core routers,

67

DRAFT




which in turn interconnect to the provider network border router. This border router is connected totwo ISPs (modelled with a single router since the details of the ISP network are not important forthis experiment), each one are attached to a server. A datacentre owned by the network provider isattached to one of the core routers.

UE 1

UE 2

AR 1

AR 2

AR 3

AR 4

AR 5

AR 6

CR 1

CR 2

GW 1

ISP 1

ISP 2

SRV 5

SRV 6

SRV 1

SRV 2

SRV 3

SRV 4

ToR 2

ToR 1

DC GW

Small DC

Service Provider net

Data Center Gateway

User Equipment

Provider Access Router

Core Router

Provider 1 Border Router

ISP Router

Server

Top of Rack Router

Figure 26: OMEC experiment: physical systems involved in the scenario

The DIF structure of the experiment for the communication between applications in UEs andapplications in public Internet facing DIFs is shown in Figure 27. The first hop between mobilehosts and the access network is serviced by shim DIFs over WiFi, which allow higher layer DIFsand local RINA management to interact with the WiFi protocols (triggering access point scanning,association and dissociation). On top of that the mobile network DIF provides distributed mobilitymanagement across the service provider network. Finally, a public Internet DIF allows applicationsin UEs to reach applications hosted at servers outside of the service provider network.

Figure 28 shows the DIF structure for the communication between applications in UEs andapplication hosted at the service provider private cloud. A DC Fabric DIF connects all the servers inthe service provider datacentre to the datacentre border router (labelled gateway in the Figure). TwoVPN DIFs float on top of the DC Fabric DIF, each one providing private access to the applicationsof a different company. These VPN DIFs span all the way to the UEs when those need to connectto applications hosted at the provider’s datacentre.

The experiment will analyse the following scenario:

1. Once a given mobile host has joined the Mobile Network DIF, the rina-echo-time application

68

DRAFT




UE Access 1 Core 1 Gateway

ISP1 Server6 Mobile network DIF

Internet DIF

DAF (rina-tgen or rina-echo-time)

Shim DIF WiFi Shim DIF Eth Shim DIF Eth

Shim DIF Eth Shim DIF Eth

UE A1

A4

A2 C1

GW

C2

Mobile Network DIF

I1

UE GW

S1

S2 I2

Internet DIF

A3

A5

A6

DC

Figure 27: OMEC experiment: DIF configurations: UE to server on the public Internet

69

DRAFT




UE Access 2 Core 1 DC Gateway

ToR 1 Server 1

Mobile network DIF

Enterprise 1 VPN DIF

DAF (Any demo app)

Shim DIF WiFi Shim DIF Eth Shim DIF Eth Shim DIF Eth Shim DIF Eth

S2

GW

S4 Enterprise 2 VPN DIF

DC Fabric DIF

S1

UE 1

GW

S3 Enterprise 1 VPN DIF

UE 2 S2

S1

S3

S4

GW

ToR1

ToR2

Figure 28: OMEC experiment: DIF configurations: UE to server on the provider’s cloud

70

DRAFT




in the UE will request a flow to a rina-echo-time server hosted in one of the servers outsideof the provider’s network.

2. Then the DIF Allocator will start the search for the DIF that brings to the rina-echo-timeserver, locating it and figuring out that the destination application is available through thepublic Internet DIF.

3. After that the DIF Allocator instance at the UE will create an IPC Process and instructs it tojoin the public Internet DIF.

4. Once it has joined the flow allocation request will be passed to the public Internet DIF, whowill allocate the flow.

5. A second flow to another application hosted in a public Internet facing server will be requested(now by the rina-tgen application). Time to flow setup in this case should be lower, since themobile host is already a member of the public Internet DIF.

6. While the two applications are executing, the UE moves and changes its attachment to theservice provider’s access routers.

7. Now the rina-echo-time client requests a flow to a rina-echo-time server hosted at theprovider’s private cloud.

8. The DIF Allocator will look for the destination application, and will realise it is availablethrough the Enterprise 1 VPN DIF.

9. Then the DIF Allocator instance at the UE creates an instance of an IPC Process and instructsit to join the Enterprise 1 VPN DIF (requires authentication), who later processes the flowrequest and allocates the flow.

The experiment will be carried out with a varying number of concurrent requests, starting froma single one (to verify the correct operation of all the procedures involved in the scenario). Wewill measure how the increased load in the Flow Allocator increases its response time, and howhandover performance is degraded. This data will measure the maturity and performance of theIRATI implementation - not of RINA per se, but this is also an important goal of IRATI.

5.4.6 Planning

We foresee the milestones reported in Table 28 in the development of the OMEC experiments.

71

DRAFT





MS 1 M18 Initial version of the experiment works with pre-allocated DIFs (no DIF Allocator) and 2 concurrentmobile hosts in local testbed

MS 2 M22 Experiment works with pre-allocated DIFs and 10 concurrent mobile hosts in w-iLab.t and VWalltestbeds (maps to ARCFIRE project milestone MS8)

MS 3 M26 Experiment works with DIF Allocator (dynamic DIF creation) and 2 concurrent mobile hosts in localtestbed

MS 5 M28 Experiment works with DIF Allocator (dynamic DIF creation) and 10 concurrent mobile hosts in w-iLab.t and VWall testbeds (maps to ARCFIRE project milestone MS9)

Table 28: Milestones for renumbering experiments

72

DRAFT




6 Experiment 4: Studying the effects of (Distributed) Denial of Ser-vice (D)DoS attacks inside and over RINA networks

6.1 Introduction and motivation

Distributed Denial of Service (DDoS) attacks make illicit use of a number of third-party computersto overwhelm one or more resources which are reachable through the Internet. DDoS attacks havematured and intensified during the last years [25], developing into an industry in which customerswanting to set up an attack can find DDoS toolkits by searching Google or, even easier, can pay afew tens of dollars per month to individuals providing DDoS attack services for hire. The growingnumber of cheap, insecure IoT devices connected to the Internet has contributed to scale up thesize of DDoS attacks, with record attacks disrupting personal websites (Brian Krebs on Security[26]) or Internet infrastructure companies providing DNS services (the Dyn attack [27]) havingbeen reported during the last months of 2016.

To launch a successful DDoS attack, first the attacker has to assemble an army of infectedcomputers that run the malicious software in charge of performing the attack, called a botnet. Oncethe botnet is ready, the attacker instructs it to attack one or more targets (websites, gaming servers,etc.) via command and control servers that control the operation of the botnet.

6.1.1 State of the art in botnet formation

There are multiple ways to infect a computer with a worm, trojan or other malicious software:spam, software download, insecure default settings, etc. As a representative example we will focusour analysis on the vulnerabilities exploited by the Mirai botnet, which participated in bot the Dynand Krebs website attacks. Mirai scans the Internet public address space looking for connecteddevices that have their Telnet and SSH ports open. It then tries a number of default passwords (setby the device vendors when they ship their products) from an internal dictionary, trying to gainaccess to the device. If successful, it downloads the botnet software and installs it into the infecteddevice (usually IoT devices such as security cameras, DVRs, etc., but also residential broadbandrouters). In particular Mirai exploits the following vulnerabilities:

• A global, public layer where all applications are reachable. The public Internet addressspace provides global reachability to all devices connected to it with a public IP address.Its address space is exposed to applications, therefore any application in the public Internetcan scan the whole Internet address space. Administrators that wish to restrict connectivityto/from applications on their sites have to configure a number of firewall rules so thatonly communication with authorised addresses is possible. Another option to restrict theconnectivity is to setup VPNs for different application groups (e.g. IP security cameras donot need to be reachable by the whole Internet), hence authentication would be required to beable to send traffic to the devices in such VPNs. However, setting up firewall rules or VPNs

73

DRAFT




is usually beyond the technical capability of most administrators in home or small companynetworks, which usually host the devices that are infected by the botnet.

• Use of well-known ports. The use of well-known ports instead of application namesfacilitates the large-scale scanning of Internet-connected devices, thus growing their chancesof being infected with the botnet malware. The risk can be mitigated by closing the remoteadministration ports (SSH, Webserver) that are usually open by default in this type of devices,or setting up firewall rules that only allow connections to these ports from within the localarea network. Again these techniques require some knowledge by the network administratoras well as his/her willingness to act.

• Use of insecure credentials. Since an attacker can reach the Internet-connected device withno authentication in the network layers, the only line of defense between him and the deviceare the credentials granting access to the device administration (in the application layer). Oneof the big problems with cheap IoT connected devices are that they usually come from thefactory with default credentials that are complicated to change: either they are embedded inthe firmware, or there are different credentials depending on the way of accessing the device(Webserver, SSH), or users just do not feel compelled to change the default password. Thissituation makes it trivial for a botnet to try to login several times using a dictionary-basedattack, until it finds the successful default password.

•  Global address space •  Addresses exposed to apps

•  No authentication

•  Well-known ports open •  No firewall rules nor VPNs

•  Insecure default administration credentials

Attacker Compromised Device

Public Internet

Figure 29: Vulnerabilities that enable malware to take control of Internet-connected IoT devices

6.1.2 State of the art of DDoS attacks

Once the botnet has infected enough computers it is ready to carry out the DDoS attack against oneor more targets. There are several types of DDoS attacks, all based on the premise of abusing theresources of the system under attack so that it can no longer operate on normal conditions (or evenhas to stop operating). Many of the attacks leverage IP address spoofing techniques, which allowthe sender to generate IP packets with fabricated source or destination IP addresses to make theattack more effective. The most popular types of DDoS attacks are:

74

DRAFT




UDP flood attacks. These attacks leverage the lack of flow control in the UDP protocol, whichmeans the receiver (system under attack) cannot limit the amount of information that the sender(attacker) is generating. Some of the attacks belonging to this category are:

• DNS reflection (or amplification) attacks. Attackers use publicly accessible open DNSservers to flood a target system with DNS response traffic. The primary technique consistsof an attacker sending a DNS name lookup request to an open DNS server with the sourceaddress spoofed to be the target address. When the DNS server sends the DNS recordresponse, it is sent instead to the target. Attackers will typically submit a request for asmuch zone information as possible to maximize the amplification effect, resulting in largeUDP packets sent to the target system. The root problem is that DNS does not authenticatethe requestor, therefore it cannot know that the source IP address was forged. Mitigationtechniques involve packet filtering at the ISP level to reject packets with source addressesnot reachable via the actual packet’s path [28]. Another mitigation technique is for the DNSserver to apply rate limiting on DNS responses, allowing a maximum number of responsesper second per IP address.

• NTP reflection attacks. This type of attacks are based on the same premises as DNSreflection attacks: use of an application protocol with optional authentication (thereforethe source IP address can be forged), that works on top of UDP (no flow control) and thatproduces a relatively large response to a small request (the size ratio of response vs. requestis called the amplification factor). What changes is the use of NTP (the Network TimeProtocol) instead of DNS. One of the NTP commands produces an answer that returns up to600 addresses of the latest servers NTP has interacted with; this can lead to amplificationfactors of up to x209 (the typical DNS amplification factor is x8). The easiest mitigationtechnique is to use NTP authentication.

GRE (Generic Routing Encapsulation) flood attacks. GRE mainly encapsulates data packetsand routes them through the tunnel to a destination network that de-encapsulates the payload packets.Sending many GRE packets with large amounts of encapsulated data may lead to extreme resourceconsumption when the victim tries to de-encapsulate them.

TCP SYN flood attacks. These attacks abuse the 3-way handshake by sending a TCP SYNpacket but never replying to the TCP-SYN-ACK with a TCP-ACK. This way they leave theconnection half-open in the server for some time (consuming resources) before it can be destroyed.The attacker sends another TCP-SYN packet before the connection is reset, therefore the totalnumber of “half-open” connections keeps growing until the target system can no longer acceptnew TCP connections. Mitigations to this attack include adjusting the TCP reset timers, allocatingmemory to the TCP connection only when the server receives the TCP ACK packet or using cookiesto check that the client is legitimate.

IP Fragmentation attacks. This type of attacks generate a number of manipulated segmentsof fragmented IP datagrams to cause problems in the reassembly process at the system under

75

DRAFT




attack (overlapping segments, segments that are too small, large number of incomplete packets,reassembled datagrams that are larger than the maximum size of an IP packet, etc.). These attackscan be mitigated by DDoS protection systems sitting at the network entry point that check forincoming packets that break IP fragmentation rules.

ICMP flood attacks. Similar to UDP flood attack, an ICMP flood overwhelms the targetresource with ICMP Echo Request (ping) packets, generally sending packets as fast as possiblewithout waiting for replies. This type of attack leverages the lack of flow control for ICMP packets.

6.1.3 State of the art of mitigation techniques of DDoS attacks

A comprehensive survey of the methods to defend against DDoS flooding attacks in the Internet isprovided in [29]. This survey classifies the different techniques to defend against DDoS floodingattacks depending on where they are deployed and enforced (close to the attacker, close to the targetor in intermediate networks) and when they are used: before, during or after the attack.

Source Network

Retail provider 1Transit provider Retail provider 2

Target Network

Target host

Compromised hosts

CPE CPEAccess router Access

router

Source-based defense mechanisms

Target-based defense mechanisms

Network-based defense mechanisms

Hybrid defense mechanisms

Figure 30: Classification of DDoS defense mechanisms depending on their deployment

Source-based techniques. Ingress/egress address filtering tries to limit the amount of trafficwith spoofed IP addresses leaving or entering the network (by checking that the IP address prefixis a valid one according to the network addressing policy). Traffic analysis samples inbound andoutbound network traffic, comparing it to models of normal traffic operation to detect potentialDDoS attacks. Reverse firewalls limit the outgoing network packets per destination, trying toprevent floods of packets to specific targets. In general source-based techniques have trouble indetecting DDoS attacks due to the very nature of the attack, since infected devices don’t generatevery high loads nor abnormal traffic patterns. There is also an economic problem regardingdeployment, since these mechanisms have a cost (memory/CPU utilisation) and do not protect thenetworks of the ones who deploy them (they do protect other networks).

Destination-based techniques. IP traceback mechanisms try to trace back the forged IPpackets to their true source address rather than the spoofed one, either using packet markingor link testing techniques. Management Information Base (MIB) analysis can be performed todetect abnormal patterns in the statistics of received ICMP, UDP and TCP packets. The MIB also

76

DRAFT




enables the modification of the router behaviour to react to the attack. Packet marking and filteringmechanisms aim to mark legitimate packets at each router on the path to the destination so thatthe target’s edge routers can filter the attack traffic. The marking can be based on informationregarding the history of known IP addresses, hop counts associated to legitimate IP addresses orpath identifiers embedded in IP packets. Packet dropping based on the level of congestion is basedon selectively dropping suspicious PDUs in routers once a certain threshold of congestion has beenreached. The problem with destination-based techniques is that they cannot detect and respond tothe attack until a significant number of the target resources are already being abused.

Network-based techniques. Route-based packet filtering extends ingress filtering to the coreInternet routers The other set of network-based techniques are based on detecting and filteringmalicious routers, based on analysing the traffic they forward and comparing it to a model of theexpected behaviour. These type of techniques require a significant number of memory and CPUresources, in addition to a large communication overhead between routers to coordinate; makingcurrent network-based approaches usually not effective nor efficient for fighting back against DDoSattacks.

Hybrid techniques. These techniques are deployed at multiple points, requiring coordinationbetween different components in order to be effective.

• Hybrid packet throttling techniques usually place packet detection close to the target andperform throttling close to the source of the attack. Aggregate-based congestion controland pushback compute maximum rates for flow aggregates and send pushback messages toupstream routers for the aggregates that are misbehaving. Upstream routers apply the sameapproach recursively. TRACK uses a port-marking module in routers that probabilisticallymarks packets with a local port-identifier. Systems under attack can use the informationin the packets to trace the attack back to the source, while upstream routers can filter themalicious packets.

• DEFensive Cooperative Overlay Mesh (DEFCOM) is a framework to enable the nodesparticipating in the DDoS defense to communicate. DEFCOM organises such nodes in apeer to peer network that follows an approximation of the underlying connectivity graph.DEFCOM classifier nodes differentiate legitimate traffic from attack traffic and are placedclose to the attack targets, while nodes that rate-limit malicious traffic are placed closer todestination.

• COSSACK leverages ingress/egress route filtering, attack signature matching and a multicastcommunication architecture to provide a target-identification and source-mitigation defensestrategy.

• Capability-based mechanisms let the destination explicitly authorise the traffic it desires toreceive. Destinations grant senders the capability of sending traffic via tokens that sendersmust embed in the packets they generate. Verification points along the path check if the

77

DRAFT




traffic is legitimate or not. These systems depend on securely granting capabilities, whichis usually addressed by authenticating the source (requiring routers to store cryptographicmaterial to validate legitimate packets).

• Active Internet Traffic Filtering (AITF) assumes the receiver accepts all the traffic andexplicitly denies the traffic that has been considered harmful. The receiver has to contact therouters of the ISPs that host misbehaving hosts so that their traffic is dropped.

Hybrid techniques are the most effective way to detect DDoS flooding attacks and mitigatingtheir effects, at the cost of having to deploy and coordinate DDoS defence mechanisms in multiplenetworks. Combining source address authentication (to prevent spoofing), capabilities and filteringcould provide the most effective and efficient solution. However, there will be a trade-off betweenperformance and accuracy in any DDoS defence solution.

6.2 Objectives

The previous sections introduced the state of the art of DDoS attacks that take place over theInternet, analysing some of the vulnerabilities that are exploited by the attackers. Ultimately,the ability to DDoS any shared resource depends on how many client actors can generate (somekind of) traffic toward it, and whether it is possible to source-control it or if it has to be sourcecontrolled at the destination. The high-level goal of this experiment is to investigate how the RINAenvironment is different from the Internet when it comes to DDoS attacks. Is it more resistant toDDoS attacks? Are there similar vulnerabilities to what we see in the Internet? The experimentwill also explore different configurations of RINA-based operator networks that may have differentlevels of effectiveness in preventing and mitigating DDoS attacks.

In this experiment we will focus on abusing the resources of properly-operating IPCPs. Clientscan cause the generation of a lot of management traffic by requesting flows (hitting directories andDIF allocators), generating data traffic via an established flow, or exhausting remote resources by(at a rate that can still be serviced) creating an increasing number of DIFs or flows within a DIF toa shared destination and never removing them. The number of attacking clients and the scale of theshared receiver(s) being attacked determines how serious each of these can be.

Our hypothesis is that the following properties of RINA can be leveraged to construct networksthat are less prone to the effects of DDoS attacks than the Internet is.

• Enrollment enforces explicit association control. In the internet, any pair of addressesis “already-connected” with respect to the ability to send packets to each other. The onlypairwise control is what can be imposed via routing redirection or firewalls after the fact,but neither stops the senders, it just protects the receiver. In a RINA network joining aDIF requires authentication. Before an application instance is able to send data to anotherapplication instance a flow needs to be allocated, allowing the DIF that allocates the flow toperform access control and police flow allocation requests.

78

DRAFT




• Use of application names. DIFs do not expose addresses to the applications using them. Nowell-known ports are used, each application has a location-independent application namethat identifies it.

• Congestion control at lower layers enables earlier detection and response. In the Internetcongestion control is only end-to-end at a per flow level. Hence, congestion caused by theaggregate traffic from multiple flows (like in a DDoS attack) can only be detected end-to-end.With RINA congestion control happens at each layer, with lower layers reacting to congestiondue to the aggregated traffic of multiple flows at higher layers [30]. Therefore in RINAnetworks it should be possible to detect abnormal volumes of traffic in lower layers closer tothe attackers, whereas in the Internet this can only be done close to the target of the attack.

The experiment will also try to provide answers on how more difficult is to discover vulnerableapplications in RINA networks compared to the public Internet. A bot in a RINA network hasno address space to scan, nor a list of well-known ports to probe: all it can try to do is to requestflow allocations to an application name. Therefore the namespace to “scan” is an applicationnamespace, which in most cases has a much greater scope than the public Internet address space.Insecure IoT devices such as the ones exploited in the DDoS attacks reported in the previoussections would register their “remote administration application” to one or more DIFs. Assumingthe same application protocols are used (Webserver and SSH), each IoT device would register twoApplication Entities (one for each protocol). The attacking bot would have to guess the applicationnames, and avoid blocking of the Allocate Request by the DIF Allocator (if allocating the flowfrom another DIF) or the Flow Allocator at the gateway/CPE of the user network.

6.3 Experiment scenario

Experiment 4 will investigate two general scenarios. In both scenarios we will consider 4 operatorswhich provide a variety of IPC services to end customers, all interconnected via a wholesale transitprovider. The first scenario, depicted in Figure 31, assumes a single top level DIF spanning allcustomers and operators where all applications reside. This configuration is equivalent to the publicInternet scenario, in which most applications are in a single top-layer network. Different configu-rations of attackers and targets will be investigated (geographically concentrated vs. distributedattackers, geographically concentrated vs. distributed targets). The second scenario, depicted inFigure 32, assumes multiple top-level DIFs supporting a variety of applications: security monitor-ing, customer VPNs, e-mail and web-surfing, video streaming, etc. In this scenario applications aresegmented across various DIFs, therefore hosts infected with bots will belong to different DIFs.Bots need to join different DIFs to mount an attack with the same scale as in the first scenario.

Figure 33 shows the DIF models for Experiment 4. Customer DIFs will have hosts directlyconnected to customer border routers (labelled CPEs in the Figure). Service provider networks aremodelled using a simplified version of the converged operator network design provided in D2.2 [2].

79

DRAFT




Transit provider

Provider 3

Provider 1

Provider 2

Provider 4

Customer

Customer

Customer Customer

Customer

Customer Customer Customer

Customer


Customer

Customer Customer

Customer

Customer Customer

Public Internet DIF

Target AP Bot AP DIF Legend

Public Internet DIF

Customer DIF

CPE

Compromised Host Transit

BorderProvider Border

Transit Border

Provider Border

Internal DIFs

. . . PtP DIF PtP DIF PtP DIF PtP DIF PtP DIF PtP DIF

Access Router

DIF

Internal DIFs Internal DIFs

Access Router

PtP DIF

CPE

Customer DIF

Target Host

Figure 31: Experiment 4 scenario 1: single top-level DIF (public Internet) configuration

Transit provider

Provider 3

Provider 1

Provider 2

Provider 4

Customer

Customer

Customer Customer

Customer


Customer


Customer

Customer Customer

Customer

Customer Customer

Security monitoring DIF

Video streaming DIF

Web DIF

Customer VPN DIF

Target AP Bot AP DIF Legend

Security Monitoring DIF

Customer DIF

CPE

Compromised Host Transit

BorderProvider Border

Transit Border

Provider Border

Internal DIFs

. . . PtP DIF PtP DIF PtP DIF PtP DIF PtP DIF PtP DIF

Access Router

DIF

Internal DIFs Internal DIFs

Access Router

PtP DIF

CPE

Customer DIF

Target Host

Customer VPN DIF

Figure 32: Experiment 4 scenario 2: multiple top-level DIFs configuration

80

DRAFT




The model consists of one or more DIFs used by applications (the public Internet one or multipleapplication specific DIFs) running on top of an internal DIF.

CPE

. . .

Host

Host

Host

Model for the customer DIFs

. . .

. . .

. . .

CPE

CPE

CPE

CPE

CPE

CPE

PAR

PBR

PBR

Model for the Provider Internal DIF

Model for the Transit Provider Internal DIF

TBR

TBR TBR

TBR

. . .

. . .

. . .

. . .

. . .

. . .

. . .

. . . . . .

. . . . . .

. . .

. . .

Model for the Public Internet DIF

Host

Host

Host

Host

Host

Host

Host

Host

CPE

CPE

PBR

PBR

PBR PBR

PBR PBR

PBR

PBR

TBR

TBR

TBR

TBR

PAR

CPE

CPE

CPE = Customer Premises Equipment

PAR = Provider Access Router

PBR = Provider Border Router

TBR = Transit Provider Border Router

N-1 DIF

N-1 flow IPC Process

Figure 33: Models for the different DIFs present in the DDoS experiments

This configuration is adequate for the DDoS experiment, since it will allow us to look at thebehaviour of aggregate flows in the internal DIF of the provider while DDoS attacks take place.The internal DIF aggregates traffic of customer CPEs via access routers into the provider borderrouters, where the higher-level DIFs transporting application traffic are available. The transit DIF ismodelled as four IPC Processes in a full-mesh configuration. Finally, the public Internet DIF floatson top of the other DIFs, providing connectivity to applications running in hosts via CPEs, ProviderBorder Routers and Transit provider border routers. The model for the application-specific DIFs inscenario 2 is not shown because it is similar to the public Internet DIF but with each individualapplication specific DIF having a smaller scope.

The experiments in WP4 will be carried out over two configurations, labelled small-scale andlarge-scale. The systems involved in the small-scale configuration is depicted in Figure 34. 41systems are organised in two service providers, each one servicing 4 customers with 3 hosts. Thetwo providers are interconnected by a transit provider that has 3 border routers. A diagram of

81

DRAFT




Host

Host

Host

CPE

Host

Host

Host

CPE

AR

Host

Host

Host

CPE

Host

Host

Host

CPE

AR

PBR TBR TBR

TBR

PBR

Host

Host

Host

Host

Host

Host

Host

Host

Host

Host

Host

Host

CPE

CPE

CPE

CPE

AR

AR

Cus

tom

er 1

Provider 1

Transit Provider

Provider 2

Cus

tom

er 2

C

usto

mer

3

Cus

tom

er 4

Cus

tom

er 5

C

usto

mer

6

Cus

tom

er 7

C

usto

mer

8

CPE = Customer Premises Equipment

PAR = Provider Access Router

PBR = Provider Border Router

TBR = Transit Provider Border Router

Figure 34: Systems in the small scale experimental scenario

82

DRAFT




the connectivity between the systems in the large-scale configuration is shown in Figure 35. Thescenario features 4 network providers servicing 4 or 6 customers with 3 hosts each one, making atotal count of 100 systems.


Host

Provider Access Router

Provider Border Router

Transit Provider Border Router

Transit Provider

Provider 3

Provider 2

Provider 1

Provider 4

Cus

t 1

Cus

t 2

Cus

t 3

Cus

t 4

Cus

t 5

Cus

t 6

Cus

t 7

Cus

t 8

Cus

t 9

Cus

t 10

Cus

t 11

Cus

t 12

Cust 13 Cust 14 Cust 15 Cust 16

Cust 17 Cust 18 Cust 19 Cust 20

Figure 35: Systems in the large scale experimental scenario

Each scenario will be tested in two configurations: in the first one all networks will not berunning any policy specifically designed to identify and mitigate DDoS attacks. Only standard RINApolicies performing scheduling, flow control and congestion control will be used in the differentnetworks. By analysing the result of the experiment runs, we will identify the vulnerabilities thatcan be mitigated by specific policies (such as rate-limiting of flow allocation requests, or limitingthe remote applications that can discover and connect to local applications) and what informationcan be used to identify the sources of the DDoS attack. Such policies will be added to the secondconfiguration, thus allowing us to compare the complexity and scalability of the solutions deployedwith the gains obtained with regards to the KPIs described in Section 6.4. The experiment willfocus on two configurations for the target application. The first one, the standalone target, willtest a single instance of the target application, in a deployment equivalent to a DDoS attack to asimple personal website (such as the attack to the Krebs blog [26]). The second target applicationconfiguration, called distributed target, features a deployment equivalent to the attack to the DynDNS infrastructure [27]. In such configuration there are application instances that can do the task of

83

DRAFT




he Application (worker APs) and many request handling AP instances spread around the network.Allocates to the distributed application are forwarded to the “nearest” DAP member. If the zombiedistribution showed some locality, such as mostly from North America, etc. this would allowrequests to be distributed to request handling APs so that some of them would be under attack andothers would not be. The request handling APs could forward valid requests, i.e. those that getthrough CACEP authentication, to one or more of Worker APs that do the real application work ormay do it themselves.

8

Scenario 1 (Public Internet DIF)

Standard policies

Standalone target Botnet in all operators

Botnet in 2 operators

Botnet in 1 operator

Distributed target

Botnet in all operators



DDoS-specific policies




Distributed target




Scenario 2 (App-specific DIFs)

Standard policies




Distributed target




DDoS-specific policies




Distributed target




DDoS Experiments

DIF Configurations Type of policies Type of target Distribution of bots

Figure 36: Categorisation of DDoS experimental scenarios

Three botnet configurations will be tested for each type of target application deployment: onein which all the bots are located in a single provider, a second one in which the bots are spreadacross two providers and a final one in which the bots are evenly distributed across all the providerspresent in the scenario. Figure 36 summarises all the scenarios that will be tested under Experiment4. Different runs with a growing number of bots will be performed to measure how the attacks scale

84

DRAFT




as the number of bots increases.Several countermeasures to DDoS attacks for RINA networks will be explored in the scenarios

with DDoS-specific policies. We anticipate the following ones:

• Isolation of flow allocation traffic. Flow Allocation traffic and RIB Daemon traffic can beisolated on different EFCP flows so that a DDoS attack does not impact DIF operations.

• Rate-limiting of flow allocation requests. In each system rate-limit Flow Allocation re-quests; in the directories (DIF Allocator or Flow Allocator) rate-limit the number of flowallocation requests that come from the same IPCP and/or are directed to the same IPCP.Analysing the pattern of proper flow allocate request distribution vs. the one in DDoS attacksis also key, then the DIF Allocator/ Flow Allocator can decide to block it.

• Resource allocator mitigation techniques. Resource allocators on IPCPs can collaborateto identify the traffic of the attack (based on observing the use of resources in multiple pointsat the DIF) and request IPCPs in provider routers close to the sources to discard the PDUsbelonging to the flows of the attackers.

6.4 Metrics and KPIs

The objective of this experiment is to compare the resiliency of RINA networks against DDoSattacks with that of current networks.


Difficulty in discoveringvulnerable applications

Size of application names-pace

32 (IPv4) or 128 bits (IPv6)of address space * 16 bits oflocal ports per host

Size can be designed accord-ing to different goals (e.g. tomake scanning hard)

Cost of preventing vulnera-ble application discovery

Number of rules that need tobe written in access controlsystem

Number of firewall rulesdepends on the specificscenario

At least 50% less than in theTCP/IP case

Accuracy in detecting DDoSattack

Factors that bring to falsepositives or false negatives

Depends on particular strat-egy used

Using comparable strategies,its accuracy should be higheron RINA networks

Timeliness Delay in detection, response Depends on particular strat-egy used

Using comparable strategies,detection time should belower on RINA networks

Scalability Extra state in the networkdue to DDoS defense perattacker

Depends on particular strat-egy used

Using comparable strategies,the amount of state in thenetwork per attacker shouldbe lower

Table 29: KPIs for DDoS attacks

The KPIs and metrics that will be taken into consideration for Experiment 4 are listed inTable 29. The experiment will analyse the complexity of preparing a DDoS attack, which mainly

85

DRAFT




involves assembling an army of infected computers running bots that can be commanded to attackthe target(s). We will focus on the analysis of network-related issues. We will assume thatthe identification of vulnerable computers is carried out using network scanning techniques. Inaddition to measuring the difficulty in discovering vulnerable applications, we will also analyse theeffectiveness and complexity of the countermeasures that can be utilised to prevent the attackerto gain access to such vulnerable computers. The other KPIs focus on DDoS attack detection andmitigation. Many of the metrics that can be applied to evaluate the effectiveness of a DDoS solutionsuch as attack detection accuracy, precision, delay in detection/response, performance, etc. dependon the particular DDoS mitigation technique as well as on the architecture on which it is applied(TCP/IP or RINA). The main interest is to understand how RINA relates to TCP/IP when differentRINA strategies are deployed. Some of the time, a direct comparison of measurement data willnot provide a complete picture as it may measure inefficiencies in the implementation rather thanthe potential benefits of the RINA architecture. For instance, as introduced in section 6.1.2, themost effective solutions to fight DDoS attacks (the hybrid ones) require coordination by multiplesystems within different networks. In the current Internet this coordinated response is hard torealise because it requires the deployment of functions that facilitate this coordination (frameworkssuch as DEFCOM, for instance). We will envestigate how the comprehensive layer managementarchitecture can simplify the extra functions required for coordinated DDoS defense. The KPIs andmetrics that will be taken into account for this part of Experiment 4 are listed in Table 29.

6.5 Software

The required software for this experiment is summarised in Table 30.

• IRATI implementation. It will be used to implement multiple RINA-enabled systems andrun a variety of DIFs according to the experiment configuration depicted in Section 5.6.However, some specific policies to defend against certain forms of DDoS attacks may beimplemented by WP3 and contributed to the IRATI repository (such policies include forinstance the ability to rate-limit flow allocation requests).

• Nginx web server. A version of the Nginx web server ported to the RINA API will be usedas one of the target applications to attack.

• DIF Allocator. The prototype DIF Allocator that is under development within WP3 will beused to experiment with DDoS attacks involving applications in multiple DIFs. WP4 willprovide feedback to WP3 in regards to specific DIF Allocator features that may need to beimplemented for the experiment (such as rate-limiting flow allocation requests).

• DDoS botnet application. A prototype botnet application will be developed to carry on thedetection of vulnerable systems (recruiting zombies for the attack) and to perform a numberof different DDoS attacks (flow allocation request flooding, traffic flooding). The DDoS

86

DRAFT




application will be modelled after the source code of the Mirai botnet [31], adapting it to thecapabilities and restrictions of RINA networks.

• Demonstrator. The demonstrator will be used to test the correct behaviour of the softwarebefore deploying it on FED4FIRE+ testbeds. The demonstrator allows the experimenter tocarry out multiple experiment runs in a controlled environment with enough scale (up to 100RINA-enabled Virtual Machines).




Nginx [32] Open source web server Modified 2-clause BSD

DIF Allocator [5] Locates applications over a variety of DIFs and collaborates withNMS(s) to create new DIFs

TBD

DDoS botnet application [5] Botnet for RINA networks that infects vulnerable systems andlaunches DDoS attacks

TBD


Table 30: Software required for Experiment 4

6.6 Testbeds

Experiment 4 does not plan to use wireless resources, therefore the Virtual Wall will be the maintestbed of reference. The Virtual Wall can provide access to bare metal hardware to run the RINAimplementation and the applications required for the experiment. The number of resources availableat the Virtual Wall is also adequate (the largest scenario planned for the DDoS experiments containsabout 100 systems). After having run the experiment at the Virtual Wall, we plan to use a secondtestbed to enhance the geographical scope of the tests as well as the number of resources used.The current plan is to use some of the racks available through GENI, leveraging the expertise ofBoston University in using this testbed. However, the final decision will be made at the end of year2 (month 24), since inter-testbed connectivity is expected to change and evolve as the FED4IRE+project progresses its work. A summary of the testbeds that will be used from the selection made inD4.2 [9] can be seen in Table 31.

6.7 Planning

Table 6.7 shows the planned milestones for Experiment 4.

87

DRAFT




Testbed Purpose

Virtual Wall Provides access to enough bare metal servers to carry out DDoS experiments

GENI racks Enhance geographical diversity of the experiment; enable larger experiments

Table 31: Testbeds used in experiment 4


MS 1 M18 All software for experiments with scenario 1 works on the Demonstrator, initial results using small-scale deployment

MS 2 M19 All software for experiments with scenario 1 works on the Demonstrator, initial results using large-scale deployment

MS 3 M22 Scenario 1 experiments carried out at VWall testbed using small-scale deployment (maps to ARCFIREproject milestone MS8)

MS 4 M23 Scenario 1 experiments carried out at VWall testbed using large-scale deployment

MS 5 M26 Scenario 2 experiments carried out at VWall testbed using both small-scale and large-scale deploy-ments

MS 6 M28 Multi-testbed experiment (tentative ones are VWall and GENI) carried out using large-scale deploy-ment (maps to ARCFIRE project milestone MS9)

Table 32: Milestones for DDoS experiments

88

DRAFT




7 Conclusion

This deliverable details the experimentation plans for ARCFIRE to be conducted during the secondpart of the project. Four main experiments were drafted, spanning different aspects such asmanageability, scalability, robustness and efficiency, with relevant KPIs identified for each of theseaspects per use case. On the basis of the use case, testbeds were selected for each experimentand the necessary software to execute the experiments were identified. A planning for eachof the experiments has been document to assess progress. As such, this document will guidethe experimenters to achieve meaningful results and aid them towards a successful and timelyconclusion.

89

DRAFT




References

[1] ARCFIRE consortium. (2016, September) H2020 arcfire deliverable d2.1: Convergednetwork operational environment analysis report. [Online]. Available: http://ict-arcfire.eu

[2] ——. (2016, December) H2020 arcfire deliverable d2.2: Converged service provider networkdesign report. [Online]. Available: http://ict-arcfire.eu

[3] International Telecommunication Unit, “Management framework for open systemsinterconnection (osi) for ccitt applications,” ITU-T Recommendation X.700 (09/92), March1992. [Online]. Available: www.itu.int/rec/T-REC-X.700

[4] L. Andrey, O. Festor, A. Lahmadi, A. Pras, and J. Schonwalder, “Survey of snmp performanceanalysis studies,” International Journal of Network Management, vol. 19, pp. 527–548, 2009.

[5] IRATI Github site. [Online]. Available: https://github.com/irati/stack

[6] Rina traffic generator. [Online]. Available: https://github.com/IRATI/traffic-generator

[7] Rina demonstrator github site. [Online]. Available: https://github.com/IRATI/demonstrator

[8] Rumba measurement framework. [Online]. Available: https://gitlab.com/arcfire/rumba

[9] ARCFIRE consortium. (2016, December) H2020 ARCFIRE deliverable D4.2: Experimentalinfrastructure available for experimentation report. [Online]. Available: http://ict-arcfire.eu

[10] V. Ishakian, J. Akinwumi, F. Esposito, and I. Matta, “On supporting mobility and multihomingin recursive internet architectures,” Computer Communications, vol. 35, no. 13, pp. 1561–1573, 2012.

[11] F. Giust, L. Cominardi, and C. Bernardos, “Distributed mobility management for future5g networks: overview and analysis of existing approaches,” IEEE IEEE CommunicationsMagazine, vol. 53, no. 1, pp. 142–149, January 2015.

[12] PRISTINE Consortium. (2016, June) FP7 PRISTINE, deliverable D4.3: Final specificationand consolidated implementation of security and reliability enablers. [Online]. Available:http://ict-pristine.eu

[13] E. C. Bormann, “Robust header compression (rohc): Framework and four profiles: Rtp, udp,esp, and uncompressed,” IETF Standards track, RFC 3095, 2001.

[14] (2017, Feb.) The IRATI ioq3 port. [Online]. Available: https://github.com/irati/ioq3

[15] ARCFIRE consortium. (2016, December) H2020 ARCFIRE deliverable D3.1: Integratedsoftware ready for experiments: RINA stack, Management System and measurementframework. [Online]. Available: http://ict-arcfire.eu

90

http://ict-arcfire.eu


www.itu.int/rec/T-REC-X.700

https://github.com/irati/stack

https://github.com/IRATI/traffic-generator

https://github.com/IRATI/demonstrator

https://gitlab.com/arcfire/rumba


http://ict-pristine.eu

https://github.com/irati/ioq3


DRAFT




[16] OpenAIRInterface. [Online]. Available: http://www.openairinterface.org/

[17] D. Shinghal, M. Kunapareddy, V. Chetlapalli, V. B. James, and N. Akhtar, “Lte-advanced:Handover interruption time analysis for imt-a evaluation,” in 2011 International conferenceon Signal Processing, Communication, Computing and Networking Technologies(ICSCCN),2011.

[18] rlite github site. [Online]. Available: https://github.com/vmaffione/rlite

[19] jFed website. [Online]. Available: http://jfed.iminds.be

[20] Quagga routing suite. [Online]. Available: http://www.nongnu.org/quagga/

[21] S. Leon, J. Perello, D. Careglio, E. Grasa, M. Tarzan, N. Davies, and P. Thompson., “Assuringqos guarantees for heterogeneous services in rina networks with δq,” Proceedings of the 6thWorkshop on Network Infrastructure Services as part of Cloud Computing (NetCloud), 2016.

[22] B. Carpenter, R. Atkinson, and B. Flink, “Renumbering still needs work,” IETF Informationaldraft, RFC 5887, May 2010.

[23] D. Leroy and O. Bonaventure, “Preparing network configurations for renumbering,” Inter-national Journal of Network Management, vol. 19, no. 5, pp. 415–426, September/October2009.

[24] C. Buyukkoc. (2016, Mar.) Edge definition and how it fits with5g era networks. [Online]. Available: http://sdn.ieee.org/newsletter/march-2016/edge-definition-and-how-it-fits-with-5g-era-networks

[25] Verisign. (2016, October) Verisign distributed denial of service trends report, q3 2016.[Online]. Available: https://www.verisign.com/assets/report-ddos-trends-Q32016.pdf

[26] B. Krebs. (2015, September) Krebs on security hit by record ddos. [Online]. Available:https://krebsonsecurity.com/2016/09/krebsonsecurity-hit-with-record-ddos/

[27] A. Technica. (2016, October) How one rent-a-botnet army of cameras, dvrs causedinternet chaos. [Online]. Available: http://arstechnica.com/information-technology/2016/10/inside-the-machine-uprising-how-cameras-dvrs-took-down-parts-of-the-internet/

[28] P. Ferguson and D. Senie, “Network ingress filtering: Defeating denial of service attackswhich employ ip source address spoofing,” IETF Network Working Group, Best CurrentPractice 38, May 2000.

[29] S. T. Zargar, J. Joshi, and D. Tipper, “A survey of defense mechanisms against distributeddenial of service (ddos) flooding attacks,” IEEE communications surveys and tutorials, vol. 15,no. 4, pp. 2046–2069, 2013.

91

http://www.openairinterface.org/

https://github.com/vmaffione/rlite

http://jfed.iminds.be

http://www.nongnu.org/quagga/

http://sdn.ieee.org/newsletter/march-2016/edge-definition-and-how-it-fits-with-5g-era-networks

http://sdn.ieee.org/newsletter/march-2016/edge-definition-and-how-it-fits-with-5g-era-networks

https://www.verisign.com/assets/report-ddos-trends-Q32016.pdf

https://krebsonsecurity.com/2016/09/krebsonsecurity-hit-with-record-ddos/

http://arstechnica.com/information-technology/2016/10/inside-the-machine-uprising-how-cameras-dvrs-took-down-parts-of-the-internet/

http://arstechnica.com/information-technology/2016/10/inside-the-machine-uprising-how-cameras-dvrs-took-down-parts-of-the-internet/

DRAFT




[30] P. Teymoori, M. Welzl, S. Gjessing, E. Grasa, R. Riggio, K. Rausch, and D. Siracussa,“Congestion control in the recursive internetwork architecture (rina),” IEEE ICC 2016, NextGeneration Networking and Internet Symposium, 2016.

[31] (2016) Mirai botnet source code. [Online]. Available: https://github.com/jgamblin/Mirai-Source-Code

[32] Nginx website. [Online]. Available: http://hg.nginx.org/nginx.org

92

https://github.com/jgamblin/Mirai-Source-Code

https://github.com/jgamblin/Mirai-Source-Code

http://hg.nginx.org/nginx.org

DRAFT - ARCFIRE projectict-arcfire.eu/wp-content/uploads/2015/12/arcfire_D4.3.pdf3 Experiment 1: Management of multi-layer converged service provider networks16 ... 6 Experiment 4:

Documents