Treatment Frameworkkth.diva-portal.org/smash/get/diva2:1299705/FULLTEXT01.pdfAbstract The middlebox architecture is long known for its inharmonious presence within the Internet architecture.

EXAMENSARBETE AVANCERAD NIVÅ, 30 HPSTOCKHOLM, SVERIGE 2019

Treatment FrameworkTraffic Steering via Source-Routing in SDN for Service Function Chaining

HÜSEYIN KAYAHAN

KTHSKOLAN FÖR ELEKTROTEKNIK OCH DATAVETENSKAP

Treatment Framework

Traffic Steering via Source-Routing in SDN for Service Function Chaining

Huseyin Kayahan

Master of Science Thesis

Communication SystemsSchool of Electrical Engineering and Computer Science

KTH Royal Institute of Technology

Stockholm, Sweden

10 December 2018

Examiner: Peter Sjodin

© Huseyin Kayahan, 10 December 2018

Abstract

The middlebox architecture is long known for its inharmonious presence within theInternet architecture. Network functions realized in middleboxes are inclined tointerpose end to end connections, modifying the datagram header or spawning newconnections on behalf, which renders policy enforcement challenging. Moreover, theirtight coupling with metadata makes its distributed persistence difficult, which hampersthe flexible utilization and scalable provisioning of the middlebox infrastructure resourcesunder varying loads. Existing attempts at mitigating these problems include middleboxplacement, packet tagging and metadata migration; each solving only a part of theproblem.

Investing in the extensible nature of IPv6, the Treatment Framework (TRF) exploitssource routing with the flavor of a discretionarily classifiable address space. Datagramstraverse the treatment domain with an extension header pushed and popped at thedomain’s edges, for which forwarding takes place based on the information encodedwithin. The forwarding mechanics that leverage SDN consists of one match and threeOpenFlow actions implementation, whereby TRF obviates the need for an underlyingtransport. Customizable address space allows providers to tailor routing aggregation totheir middlebox farms topology, reducing the number of flow rules in the core to pre-installable sizes.

Middleboxes in a treatment domain match traffic to the respective local policy basedon the information encoded in the extension header. Extension headers are native toIPv6 and defined by standards, hence the middlebox modification problem is addressedwithout requiring alteration nor visibility into proprietary code. The framework resolvesthe policy enforcement problem altogether and allows asymmetric service chaining.While eliminating the flow setup time in the core, the framework’s footprint at ingressthat push the extension header can get heavy with respect to flow churn rate.

i

Sammanfattning

Det har lange varit kant att arkitektur baserad pa mellanliggande utrustning, s.k.middlebox-arkitektur, inte alltid rimmar val med Internet i ovrigt. Natverksfunktionersom implmenteras i middleboxar tenderar att leda till olika typer av forbindelser,modifierande av pakethuvuden eller nya uppkopplingar, vilket medfor att det blirutmanande att verkstalla olika typer av policy. Middlebox-losningar ar dessutom tattkopplade till olika typer av metadata vilket innebar svarigheter for dess distribueradefortlevnad och hindrar ett flexibelt utnyttjande och skalbar utbyggnad middlebox-resurser under varierande trafiklast. Befintliga forsok att mildra sadana problem inkluder-ar placering av mellanliggande utrustning, paketmarkning och migration av metadata;vart och ett av dessa loser endast delar av problemet.

TRF (Treatment Framework) drar nytta av den utbyggbarhet som finns i IPv6och anvander vagval som styrs helt fran avsandaren tillsammans med diskretionaradressrymd. IP-paketen traverserar en behandlings-doman och ett utokat pakethuvudanvands inom domanen. Det utokade pakethuvudet laggs till nar paketet ar pa vag in idomanen och tas bort nar paketet lamnar domanen. Inom domanen anvands informationi det utokade pakethuvudet for att styra vidarebefordringen av paketet. Mekanismernafor vidarebefordring av paket anvander sig av SDN och bestar av en match-operation ochtre OpenFlow-atgarder, varmed TRF kringgar behov av en underliggande transport. Enanpass-ningsbart adressrymd gor det mojligt for leverantorer att skraddarsy vagvalsagg-regering till sin middlebox-losning, vilket gor att antalet trafikregler i karnan av derasnat kan reduceras till forinstallbara storlekar.

En middlebox-losning i en behandlings-doman matchar trafik mot respektive lokalpolicy baserad pa information kodad i det utokade pakethuvudet. Tekniken med utokatpakethuvud ar inbyggd i IPv6 och standardiserad vilket gora att paketmodifiering i enmiddlebox kan adresseras utan att krava andringar eller insyn i proprietar programvara.Ramverket loser problemet med verkstalla olika typer av policy i sin helhet ochmedger skapandet av assymtriska kedjor av paketbehandlingstjanster. Tiden for att sattaupp paketfloden i karnan av natet kan elimineras, men det fotavtryck som ramverketger i ingressdelen av domanen, dar det utokade pakethuvudet skapas och laggs tillinkommand paket, kan bli betydande med avseende pa flodeshastigheten.

iii

Acknowledgements

I would like to thank my examiner Peter Sjodin and supervisor Markus Hidell for theirvaluable feedback, putting this thesis into shape.

I would also like to thank to OVS and FastClick communities, especially TomBarbette for his timely and invaluable feedback and Georgios Katsikas for his previouswork and feedback, without whom the implementation of this work would become anightmare.

Finally, I would like to express my sincere gratitude and appreciation to my familyand friends, who helped me to keep a high motivation.

v

Contents

1 Introduction 11.1 Problem Context . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21.2 Problem Description . . . . . . . . . . . . . . . . . . . . . . . . . . . 31.3 Purpose . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41.4 Goals . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51.5 Research Methodology . . . . . . . . . . . . . . . . . . . . . . . . . . 61.6 Structure of this thesis . . . . . . . . . . . . . . . . . . . . . . . . . . 6

2 Background 72.1 Middlebox — Essential yet Unwelcome . . . . . . . . . . . . . . . . . 9

2.1.1 From Middlebox to Virtual Appliance . . . . . . . . . . . . . . 112.1.2 Statefulness in Autonomy . . . . . . . . . . . . . . . . . . . . 122.1.3 State Taxonomy . . . . . . . . . . . . . . . . . . . . . . . . . 162.1.4 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17

2.2 Programmable Networks . . . . . . . . . . . . . . . . . . . . . . . . . 182.2.1 Active Networking . . . . . . . . . . . . . . . . . . . . . . . . 182.2.2 Participatory Networking . . . . . . . . . . . . . . . . . . . . . 202.2.3 Software Defined Networking . . . . . . . . . . . . . . . . . . 222.2.4 NFV and SDN Interplay . . . . . . . . . . . . . . . . . . . . . 24

2.3 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31

3 Treatment Framework 323.1 Framework Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . 323.2 VNF/Middlebox Presence . . . . . . . . . . . . . . . . . . . . . . . . 333.3 Addressing and Routing . . . . . . . . . . . . . . . . . . . . . . . . . 343.4 Realizing Treatment Policies . . . . . . . . . . . . . . . . . . . . . . . 35

3.4.1 TRH Construction . . . . . . . . . . . . . . . . . . . . . . . . 353.4.2 The Treatment Header . . . . . . . . . . . . . . . . . . . . . . 353.4.3 Forwarding and Classification . . . . . . . . . . . . . . . . . . 37

3.5 Bring Your Own State . . . . . . . . . . . . . . . . . . . . . . . . . . . 383.6 Use Case: IDS-NAT Service Chain . . . . . . . . . . . . . . . . . . . . 38

vii

viii CONTENTS

3.6.1 Preconditions . . . . . . . . . . . . . . . . . . . . . . . . . . . 393.6.2 Address Space Classification Schema . . . . . . . . . . . . . . 393.6.3 Treatment Policy . . . . . . . . . . . . . . . . . . . . . . . . . 393.6.4 The Treatment Process . . . . . . . . . . . . . . . . . . . . . . 40

3.6.4.1 Flow Setup . . . . . . . . . . . . . . . . . . . . . . . 403.6.4.2 Flow Treatment . . . . . . . . . . . . . . . . . . . . 403.6.4.3 Flow Termination . . . . . . . . . . . . . . . . . . . 41

4 Prototype Implementation 434.1 New OpenFlow Match and Actions in RYU and OVS . . . . . . . . . . 434.2 Controller Applications . . . . . . . . . . . . . . . . . . . . . . . . . . 444.3 Packet Analyzer . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46

5 Analysis 475.1 The Analysis Environment . . . . . . . . . . . . . . . . . . . . . . . . 475.2 Experiments and Measurement Methods . . . . . . . . . . . . . . . . . 50

5.2.1 Test Cases . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 505.2.2 Evaluation Criteria . . . . . . . . . . . . . . . . . . . . . . . . 515.2.3 Evaluation Methods . . . . . . . . . . . . . . . . . . . . . . . 52

5.3 Forwarding Performance . . . . . . . . . . . . . . . . . . . . . . . . . 555.4 Controller Performance . . . . . . . . . . . . . . . . . . . . . . . . . . 565.5 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 59

6 Conclusions and Future Work 606.1 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 606.2 Sustainability and Ethical Aspects . . . . . . . . . . . . . . . . . . . . 626.3 Future Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62

6.3.1 Opportunities and Hardships . . . . . . . . . . . . . . . . . . . 636.3.2 Insights and Suggestions . . . . . . . . . . . . . . . . . . . . . 64

Bibliography 65

List of Figures

1.1 Three dimensions of ideal service chaining. . . . . . . . . . . . . . . . 21.2 Proposed architecture overview. . . . . . . . . . . . . . . . . . . . . . 4

2.1 Autononomous internetworking model . . . . . . . . . . . . . . . . . . 82.2 Scalability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 132.3 Flexibility . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 152.4 Active Node components (Adapted from Figure 1 of [1]) . . . . . . . . 192.5 Operation of outbound NAT function in ANANTA . . . . . . . . . . . 212.6 Heteronomous internetworking model . . . . . . . . . . . . . . . . . . 242.7 State Management Frameworks . . . . . . . . . . . . . . . . . . . . . . 252.8 Network Service Header . . . . . . . . . . . . . . . . . . . . . . . . . 302.9 Service Chaining via NSH . . . . . . . . . . . . . . . . . . . . . . . . 31

3.1 TRF components . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 333.2 Treatment Header v1 Layout . . . . . . . . . . . . . . . . . . . . . . . 363.3 Setting up the flow . . . . . . . . . . . . . . . . . . . . . . . . . . . . 413.4 Packet treatment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42

4.1 TRAPP Flowchart . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45

5.1 Testbed layout . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 495.2 Packet forwarding rate . . . . . . . . . . . . . . . . . . . . . . . . . . 555.3 End-to-end Delay . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 565.4 Flow Setup Time . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 575.5 Partial (38.85%) flamegraph of the Ryu controller. . . . . . . . . . . . . 58

ix

List of Acronyms and Abbreviations

BGP Border Gateway Protocol

BYOS Bring Your Own State

DPDK Data Plane Development Kit

EoR End of Rack

EOL End Of Life

EOL End Of Support

h-node Heteronomous Node

NAT Network Address Translation

NF Network Function

NFV Network Function Virtualization

NSH Network Service Header

OSPF Open Shortest Path First

SDN Software Defined Networking

TD Treatment Domain

TE Treatment Edge

TLV Type Length Value

ToR Top of Rack

TP Treatment Policy

TRF Treatment Framework

x

LIST OF ACRONYMS AND ABBREVIATIONS xi

TRH Treatment Header

OVS Open Virtual Switch

VNF Virtual Network Function

Chapter 1

Introduction

Massive amounts of IP traffic is occurring globally on the Internet and the mainbeneficiaries of this phenomenon are the content providers that generate this trafficand the end users that consume it. Those who build and maintain infrastructuresfor delivering this traffic on the other hand, namely the operators, are struggling tokeep their business commercially feasible. This is primarily because the flat ratecharges remain the same while the load incurred by the customers keeps increasing [2].Eventually the operators turn to non-orthodox ways of making profit such as providinghigh-quality access to streams or services for an extra fee or block competitor content[3, 4, 5], bringing an end to net neutrality.

Meanwhile, to remain competitive, operators facilitate middleboxes for optimi-zingkey performance areas such as capacity, delay and security. Middlebox infrastructureaccounts for a considerable portion of the operator’s expenditure so its effectiveutilization is vital to increase profitability in this particular case of a two-sided market.Therefore the operators are in a continuous pursuit of optimum resource utilization,where the maximum possible amount of traffic can be accommodated on the middleboxinfrastructure with the least administrative labor.

One prominent way to employ a middlebox infrastructure, commonly known asnetwork function chaining or service chaining, enables operators to subject particulartraffic to a sequence of middleboxes in a desired order. The procedure necessitatespackets to follow along paths for which middlebox topology is the decisive vectorin determination, rather than the distance to packets’ destination address. Giventhe extensive flexibility in composing forwarding paths, operators shift interest fromconventional traffic engineering toward Software Defined Networking (SDN). Steeringtraffic by means of SDN is certainly promising for a more resource-friendly servicechaining, however entangles the inherent challenges of middlebox architecture and SDNwhich this thesis aims to explore.

1

2 CHAPTER 1. INTRODUCTION

1.1 Problem ContextThe ideal service chaining takes place when the traffic is steered through optimallyprovisioned resources for treatment, while its set-up and maintenance requires the leasthuman or computer involvement. As visualized in Figure 1.1, achieving this ideal isoften a matter of trade-offs between the three dimensions; traffic steering, resourceutilization and management complexity.

Figure 1.1: Three dimensions of ideal service chaining.

Resource Utilization: Just like a bare-metal server, middleboxes accommodateresources like processor and memory. Unlike a bare-metal however, they are purpose-specific enclosures in complete control of those resources; they sit idle if underutilizedand another network function in need of processor or memory resources can notmake use of them. Virtualizing middleboxes enables elasticty, allowing an operator tolaunch or destroy virtual network function instances across bare-metals, taking availableprocessor, memory and bandwidth resources into consideration.

1.2. PROBLEM DESCRIPTION 3

Traffic Steering: Elasticty is especially valuable if it can be achieved irrespective of thenetwork entry and exit points of the traffic to be processed. Traffic engineering and SDNare the prominent approaches to steering traffic from any point in the network throughthe middlebox infrastructure. While avoiding the administrative labor of physicallychaining middleboxes to set-up a service chain, it enables flexibility perks such asasymmetric flow routing which can elevate elasticity and load balancing.

Traffic engineering is commonly carried out by means of tunneling, which isthe building block of an overlay transport that enables traffic steering. However,tunneling may require significant human intervention at various levels such as planningand configuration, increasing management complexity. Some of this burden can beoffloaded to a controller in SDN, or instead, traffic steering can solely be performedbased on flow-rules, avoiding overlay transport altogether. On the downside, bothapproaches are prone to excessive controller invocations, lengthy offline computationsand prolonged online convergence, increasing con-troller overhead.

Management Complexity: Now that elasticity, traffic steering and load-balancingwork in concert, the middlebox infrastructure can scale-out per increasing load andbalance or re-route flows; then re-route flows from and terminate underutilized instancesto scale-in as the load decreases. However the orchestration of this routine has inevitableimplications in the management dimension. Furthermore, some packet processing isstateful, requiring a flow to be processed by the very middlebox that has generated andis maintaining the state for the flow. This significantly encumbers traffic steering suchas the inability to asymmetrically route a flow, or inability to re-route without sacrificingprocessing validity. State migration restores flexibility in traffic steering to facilitate theresource utilization tenets, however introduces an additional management layer for statetransfer that middleboxes have to interface, which further complicates management.

1.2 Problem Description

The mainline research covered in Section 2.2 that study the ideal service chaining,usually focus closely on only two of the dimensions; either omitting, remainingagnostic or making drastic sacrifices in the third dimension. Those that approach trafficsteering for the sake of better resource utilization often end up in a complicated andoverloaded management layer. Others that aim to improve resource utilization withclever management frameworks commonly leave traffic steering out of their scopeand just rely on the prominent means. Lastly, the widespread use of IPv4 is greatlyinfluencing the research direction to remain in this no longer sustainable protocol,encumbering researchers to re-purpose standard fields of the IPv4 header for introducing


new frameworks.

1.3 Purpose

The main purpose is to experimentally investigate source routing as a means of trafficsteering in service chaining and assess its implications on the management complexity,in pure IPv6 environment. The chosen approach is to encode source routing informationin a proprietary IPv6 extension header, and implement the required functions in SDNinfrastructure to process this header. The common components of this proposedapproach are shown in Figure 1.2.

Figure 1.2: Proposed architecture overview.

The infrastructure consists of middleboxes, an SDN controller and SDN switches.The idea is to encode source routing information to packets at an ingress switchand deliver them to respective middleboxes via steering to the switches they aredirectly connected to. The so called source routing information is a list of orderlystacked switch+middlebox identifiers, which the SDN controller compiles from servicechaining policies and encodes in the form of a proprietary header for the ingressswitches to append to packets. Departing a middlebox, the directly connected switchsets the subsequent identifier in the stack as the next-hop, based on which the coreperforms forwarding. Following the implementation of this architecture, the aim of thisinvestigation is to answer the following questions.

1.4. GOALS 5

Q1: Is source-routing a viable approach to traffic steering for service chaining?

The secondary purpose is to explore a paradigm where middleboxes operatein concert with their own proprietary controller application that undertakes statemaintenance. The controller application produces state, the controller encloses it inthe extension header, an SDN node appends the header to packets, and the middleboxconsumes the state from the very packet it is processing. The goal is to free trafficsteering from the restraints that the statefulness impose in service chaining context,answering the following question.

Q2: To what extent can the middlebox state be “mobilized”, such that a central entityproduces and appends it to the packets that are to traverse the respective state consumermiddlebox?

Combining source routing with mobilized state is the proposed framework in thisthesis, as a holistic approach at ideal service chaining. The ultimate purpose isto investigate its effectiveness, argue the benefits and the trade-offs between trafficsteering, resource utilization and management complexity, and discuss the limitationsand applicability of the framework.

1.4 Goals

Following are the goals set for validating the defined purposes of this thesis,

• Perform an in-depth background study on state of the art approaches at servicechaining, observing the hardships, opportunities and limitations.

• Introduce OpenFlow match and actions into the OVS switch implementation, inorder to push and pop the custom IPv6 extension header into the packets, as wellas to forward packets per the information encoded within.

• Implement an SDN controller application that oversees service chaining via flowrules that utilize these match and actions.

• Implement a network function with a modified packet processing pipeline that canread state from the IPv6 extension header, rather than maintaining and accessingstate locally.

• Implement an IPv6 packet generator and receiver that is capable of measuringinduced latency.


• Prototype the implementation and evaluate forwarding and controller perfor-mance with respect to throughput, end-to-end delay, flow setup time and controllerCPU utilization.

1.5 Research MethodologyThe outlined goals pave the way for prototyping the proposed framework, making itpossible to subject the framework to artificial network traffic of various packet sizes.The research then aims to perform quantitative analysis at key performance indicatorssuch as flow setup latency, packet loss, header overhead and framework induced latency,with respect to service chain length and packet generation rate.

1.6 Structure of this thesisChapter two takes a deeper dive; providing essential context for the reader to betterfollow the concepts, and examine the problem space via dissecting state of the artrelated work. Chapter three introduces the framework proposed as a solution anddemonstrates step-by-step how it applies to a use case. Chapter four presents aprototype implementation and in chapter five, findings of the evaluations performedon this implementation are presented. The sixth and final chapter concludes this thesis,discussing design decisions, limitations and future work.

Chapter 2

Background

The DARPA Internet Architecture’s primary goal was to effectively utilize interconnectednetworks in a multiplexed manner. It was built for the U.S Department of Defensehaving survivability in mind so it was designed to operate in a military contextconsidering a hostile environment of a war. Naturally, concerns such as cost effectivenessand resource accountability were pushed into the background [6], which are pivotal forthe commercial survivability of the operators.

The survivability focused mindset led to one primary design principle; avoiddisruption in communication in the event of a network failure. Circuit switching did notfit the bill as the intermediary nodes have to store per-connection state which would giveway to loss of communication in the event of a node failure. The Internet architecture’sapproach was to gather any state that is essential for the upkeep of the connectionat the participants of the communication, namely the endpoints, instead. Influencedby the earlier DARPA project ARPANET and the acceptance of its store-and-forwardtechnique, packet switching was selected as the means to multiplex. Datagram wasadopted as the entity to be transported, which does not require connection establishmentin the transient nodes, thus leaving no state behind in the nodes it traverses.

Abandonment of signaling, circuits and thus the connection state in the network infavor of survivability surfaced the need for an alternative way to path determination.ARPANET’s approach was to leave the decision of which line to use for forwarding thepacket up to the nodes by means of routing algorithms, a mechanism of store-and-forward subnetwork design. The preferred control regime for the routing protocolshappened to be, (1) distributed as the real-time computation of such algorithms forlarge networks may not be possible centrally on a single computer due to the hardwareconstraints in processor technology of the 1970s, (2) adaptive to efficiently handle nodeor link failures which is paramount for survivability [7].

7

8 CHAPTER 2. BACKGROUND

Inheriting these design decisions gave birth to what this research calls the autonomousinternetworking model illustrated in Figure 2.1. An autonomous node consists ofa control-plane where the node may run various protocols and algorithms to makeforwarding decisions, and a data-plane where the decisions are rendered to simplematch-action rules —also known as the forwarding state— for high speed packetforwarding.

Figure 2.1: Autononomous internetworking model

Horizontal cooperation: A protocol that implements a distributed adaptive algorithmruns in the control plane of an autonomous node and cooperates with the adjacent nodes.The cooperation depicted horizontally between nodes running the same protocol(s) inFigure 2.1, is in the form of information exchange that occurs periodically, upon anevent, or both. Information collected from neighboring nodes is typically collatedinto a protocol specific database —also known as the control state— which shapes theindividual forwarding decision of each node.

Vertical coordination: It is common for the administrators to configure morethan one protocol to operate simultaneously in the nodes in order to tailor the overallforwarding behavior to their expectations within an autonomous system. Shownvertically in Figure 2.1, configuration and operation of these protocols may assist, affect,or depend on each other. Therefore, their coexistence has to be carefully overseen by the

2.1. MIDDLEBOX — ESSENTIAL YET UNWELCOME 9

administrators as there is no standard mechanism to maintain the cooperation betweenthem automatically. For example, the Border Gateway Protocol (BGP) protocol maydepend on the reachability that Open Shortest Path First (OSPF) protocol would providein order to establish peering with another node running BGP several hops away. Let usassume that the direct link between the nodes goes down. OSPF would converge thetopology over an alternate path, however it may be the case that a node on the newpath might be dropping all packets per its packet filtering configuration*, which wouldimped BGP peering. Similarly, the actions taken by the spanning-tree protocol (STP)may affect OSPF, and eventually the BGP that is running along with it.†

With the increase in the number of operators due to the wide adoption of theInternet architecture, commercial concerns became more salient. After all, the Internetarchitecture has assumed survivability as its primary objective and commercial interestsare far below in the list of “secondary” goals. These concerns were primarily emanatingfrom security and performance as well as limitations, and the approach to tackling themwas packet treatment. By treating packets in a certain way an operator can achieveoptimizations that result in savings in bandwidth and processing resources, protectionfrom malicious or abusive traffic, work around the IPv4 address exhaustion, or introducenew value added services promoting competition, whereby alleviating the commercialconcerns.

2.1 Middlebox — Essential yet UnwelcomePacket treatment is defined in this thesis as any form of packet processing as a networkfunction other than routing and forwarding. Such network functionality is generallyrealized in a middlebox, a phrase coined by Lixia Zhang in 1999 referring to thecomponents that were not in the original Internet architecture. RFC 3234 defines amiddlebox as any intermediary device performing functions other than the standardfunction of an IP router on the packets of traffic flows [8], while functions that takeplace below the IP layer are excluded from this definition.

Middleboxes quickly became an anathema in the network architecture community asthey violate architectural principles of the Internet and hamper flexibility in the network.They operate on packets that are not destined to them and keep state that may directly orindirectly affect the upkeep of an end-to-end connection in case of a middlebox failure.The Internet architecture strictly avoided this as the first priority by moving essentialstate from network nodes to the end hosts. This nature of a middlebox raised a new* This problem can occur because the network and security personnel are likely to be different whichaggravates the problem.

† All three protocols can coexist on a layer 3(L3) switch which are common in datacenters. Describedproblem is especially likely when the IP is configured on a switch virtual interface (SVI).


flexibility concern called flow affinity which expresses the need that any and all trafficof a flow must be treated by the middlebox that first generated the state for it. In order tomaintain flow affinity, an operator has to forgo the flexibility of forwarding the departureand the return traffic of a flow among alternative paths independently of each other —also known as asymmetric routing— which is a survivability trait of the Internet.

Although being characterized as an autonomous node, most middleboxes do notcooperate horizontally; an update in the corporate security policy has to be reflected toall the firewalls in the network, some ACLs may even need refactoring in some firewalls.Only occasionally, one middlebox can consolidate multitude of network functions andbe subject to vertical coordination concerns. Depending on their placement, they mayhinder horizontal cooperation of neighboring autonomous nodes and might triggerrevisions to their vertical coordination. All in all they increase the human factorinvolvement, thereby exacerbating the management complexity of the autonomousinternetworking model.

Several researches attempted retrofitting middleboxes into the Internet architec-ture. One of them is the Delegation Oriented Architecture (DOA), which suggests anadditional identification layer for the nodes which renders packet treatment dependenton DNS and DHT lookups, sometimes tripling end-to end latencies [9]. A morecommon approach taken by the operators is to establish coordination and cooperationby means of automation —also referred to as orchestration— rather than extensionsto the Internet architecture. Automation introduces a management layer where theconfiguration of autonomous nodes is centrally orchestrated via protocols such asSNMP and NETCONF. Along with configuration modeled in YANG data modelinglanguage, NETCONF allows the operators to automate sequences of configuration overheterogeneous devices in a transactional manner [10].

Despite the clash of priorities —survivability in the network vs commercialsurvivability— middleboxes became an indispensable part of the Internet architec-turefor operators that drive the Internet.


2.1.1 From Middlebox to Virtual Appliance

Although RFC 3234’s definition states that a middlebox can be physical or virtual,the term is generally associated with the physical form, i.e. a standalone hardwareplatform with specialized architecture that is typically proprietary, of which theresources are strictly available to that specific network function only. In addition totheir great contribution to the management complexity explained earlier, this nature ofmiddleboxes poses another major problem; inefficient resource utilization.

Consider a scenario where a service provider has two middleboxes, both either fromthe same vendor or different. Middlebox A is an intrusion prevention system (IPS)that peaks during daytime and Middlebox B is a wan-optimizer that peaks during thenight due to off-site backups. In daytime, there exists no means for the overloaded A toutilize the idle hardware resources of B in order to cope with a load surge. In most casesa scale-out is the only solution which increases the sunk costs and the administrativelabor.

As security became vital as connected things started to grow in amount andvariety, these problems presented a major obstacle for adequate security functionimplementation, especially for small and home offices (SOHO) and small to mediumbusinesses (SMB). Firewall, intrusion detection system (IDS) or IPS, deep packetinspection (DPI), network anti-virus (NAV), content filtering and virtual private network(VPN); each being available as a separate middlebox rendered the distributed threatmanagement (DTM) expensive. Solution was sought in consolidation, which paved theway for the unified threat management (UTM) era.

UTM, as defined by IDC, is a “product that includes multiple security featuresintegrated into one box” [11], which essentially is a middlebox. Becoming the fastestgrowing segment in the security industry, UTM proved consolidation to be a success.Other network functions such as web proxies soon followed the trend of being unifiedin one middlebox.

The impact of those problems are more severe for network operators as theirinfrastructure typically house a multitude of middleboxes processing traffic at largerscales. While the global IP traffic was growing 40-50% a year, the monthly bill ofthe customers remained unchanged. This means the operators had to reduce the costof ownership by 40-50% per Gbps to keep the business profitable, but the reductionstayed below 20% in practice. Along with the inclination to commodity hardware inthe computer industry, virtualization became the key enabler for the consolidation ofmiddleboxes from different or same vendors on the same non-proprietary and affordablehardware.


Virtualization allows the hardware resources of the real machine, also known as thebare metal, to be subdivided into slices where how much of which resource the slice getsis configurable by the administrator. This configuration is done at the hypervisor, whichis a shim operating system (OS) that lies between the hardware and the slices. Users runtheir desired OSes on their slices isolated from each other, thus the term virtual machine(VM).

In 2012 at the SDN and OpenFlow World Congress in Germany, thirteen operatorsamong which are AT&T, Verizon and British Telecoms, presented a white-paper namedNetwork Functions Virtualization (NFV), calling operators worldwide for action. NFVis the concept of consolidating many network equipments such as middleboxes ontocommodity servers, switches and storage by leveraging the virtualization technology[12]. Middleboxes are nothing but a hardware and an OS that runs atop, so vendors severthe OS from the hardware and transform it into a virtual appliance (VA); a middleboxin the form of a virtual machine capable of running on a hypervisor, also referred to asVirtual Network Function (VNF).

Benefits of NFV for the operators are immense. Reduced sunk costs on hardwarethat continue to pile up upon every major feature release or end-of-life (EOL)/end-of-support(EOS) cycle. Ability to house test facilities along with the productioninfrastructure, eliminating the investment on a testbed infrastructure thus reducing thedevelopment costs. Reduced power consumption due to less hardware and improvedenergy efficiency schemes, i.e. concentrating the workload on a number of servers inorder to put the other servers into energy saving mode. Most importantly, it providesmore flexibility in resource allocation which allows the operators to scale-up or scale-out more efficiently. However the nature of network functions still pose challenges,preventing operators from availing themselves of NFV benefits to its full extent.

2.1.2 Statefulness in AutonomyVirtualization gave middleboxes a new form only; their autonomous nature remainedunchanged, inheriting the disadvantages explained in Section 2.1. Particularly the factthat they maintain state is the underlying cause for many limitations and the transitioninto VA only made these limitations relatively more manageable. This section exploressome of these key limitations in detail.

It is vital for an operator that packet treatment takes place in a resource efficientmanner, for which load distribution and scalability are pivotal intertwined instruments.An infrastructure has to take the constraints on the available resource into account andadminister the load accordingly, as well as be able to expand and shrink resources in the


event of load surges and underutilization.

Figure 2.2a depicts a typical scale out scenario. In step one, the initial packets offlow one and two end up in VA1 so in step two, VA1 creates the states for both flowsand maintains them for the duration of the flows. In step three, VA1 reaches a resourcethreshold warning so in step four, a new instance called VA2 is spawned and the nodeN1 is directed to offload traffic to this instance. Subsequent packets of flow one end upin VA2 however there is no state for this flow. Depending on the function performed,this may result in the inconsistent treatment of flow one.

Figure 2.2b shows a scale-in scenario where VA1 is underutilized so it is appropriateto shut VA2 down for saving resources. In step 1, VA1 issues the warning that it isunderutilized so that N1 can be directed to forward VA2’s traffic to VA1. Howeverflow 1 is still active and its packets are being treated by VA2. If N1 is dictated tostart forwarding the packets to VA1 at this point, it will cause inconsistent treatmentas occurred in the scale out scenario because VA1 has no state for flow 1. Thereforein step 2, N1 has to be directed not to forward new flows to VA2 which is scheduledfor shutdown in step 3, until flow 1 finalizes gracefully. Flows may stay active forlonger than 25 minutes [13] which renders waiting for flows to finalize inefficient.Yet, dictating forwarding behavior for new flows at N1 is impractical in autonomousinternetworking model as the required field(s) to match on are outside the five tuple.This inflexibility hinders not only the scalability, but also the ability to distribute theload across the infrastructure in a resource optimum manner.

(a) Scale out (b) Scale in

Figure 2.2: Scalability

As for availability, the common approach is to deploy two or more middleboxesas active/active or active/standby high availability (HA) pair/group which is inefficientand expensive. In the active/active configuration the load is balanced no matter if a


single unit can handle it alone or not and upon failure not all the state of every networkfunction may be present at the other node. In the active/standby configuration one unitstays powered on however idle, probing the active unit to take over the active role incase of failure. The time it takes for standby unit to conclude that the active one failedmay go as high as several seconds, which may result in the disruption of thousand ofconnections in an operator that processes terabits of throughput per second. Finally, theHA pairing is likely to be established over a dedicated interface which may introducephysical proximity concerns for middlebox placement.

Figure 2.3a demonstrates how violation of flow affinity can lead to anotherinconsistent treatment incident. In step one, initial departing packets of flow one arrivesat the NAT1. In step two, NAT1 instance creates the network function state for flow oneand let’s assume that the network function picked port 5000 randomly as the source portand persisted into the state, then the packets are forwarded to the network. In step three,the network renders forwarding decisions such that the packets of the return traffic offlow one are forwarded to NAT2. In step four, NAT2 instance can not find a matchingstate for packets destined to port 5000 and therefore drops the packets, causing the flowto eventually timeout. Due to the flow affinity requirement such middleboxes have tobe placed inline to the traffic on both directions, which became a problem domain of itsown.

This problem domain, the middlebox placement, becomes more challenging asoperators demand treatment of packets by several middleboxes strictly in a desiredorder, which is referred to as service chaining. Figure 2.3b demonstrates howforwarding should theoretically take place in order to realize the service chain FW →NAT defined by an administrator for the traffic class TC1. According to this servicechain, packets matching TC1 first have to be treated by a firewall, then a NATimmediately after. In this example, N1 and N2 first classify the packets sourced from10.5.10.0/24 and destined to 20.5.20.0/24 as TC1. A packet classified under TC1 departsfrom N1 in step one, it reaches N2 which forwards the packet to the FW instance. In steptwo, FW treats the packet and sends it back to N2. In step three, N2 forwards the packetback to N1 so it forwards the packet to the next network function in the service chain.In step four, NAT treats the packet and sends it back to N1, completing the treatmentdefined by the service chain. Packet is now ready at N1 in step five, to be forwarded toits destination. The hard problem that makes this theoretical flow of events impracticalis that the same packet arrives at N1 and N2 more than once, and the forwarding decisionneeds to be different for each arrival although the destination address of the packet isthe same. It may even be the case that the packet has to be forwarded to N2 for it tobe delivered to its ultimate destination, where a complete different forwarding decisionis needed at its third presence in N2. There is no native way for the autonomous N2 to


differentiate forwarding behavior in this scenario as it typically relies on the 5-tuple formaking forwarding decisions, which remained unchanged in the packet at each arrival.This problem is called forwarding ambiguity within this thesis work.

(a) Flow affinity (b) Forwarding ambiguity

Figure 2.3: Flexibility

A common workaround for middlebox placement problem is utilizing trafficengineering (TE) techniques. TE allows operators to manually override the forwardingdecisions made by shortest path algorithms in ways that would favor operator profitability,typically by means of TE tunnels. Examples include offloading traffic to underutilizedpaths or offering high service level agreement (SLA) end-to-end virtual circuits withbounded delay as a value added service. It can also help operators alleviate themiddlebox placement problem to a certain extend by steering traffic to middleboxesin desired order and maintain flow affinity. However TE depends on an interiorgateway protocol (IGP) such as OSPF and several protocols —among which aremulti-protocol label switching (MPLS), label distribution protocol (LDP), resourcereservation protocol- traffic engineering (RSVP-TE) and multiprotocol BGP (MP-BGP)— to work in concert, and requires offline path computation and planning.This dependence massively increases the vertical coordination complexity and causesexcessive resource utilization for maintaining horizontal cooperation which may resultin prolonged convergence [14].

In addition, the particular way a middlebox works can make traffic classificationdifficult, which hinders complete treatment as a service chain. For example NAT canchange the IPs and ports on which traditional way of classifying traffic relies. A WANoptimizer or a Proxy may create new connections on behalf of a flow with new five tuplein the header and hence can not be attributed to the triggering flow. Therefore N1 andN2 will fail to accurately classify such solicited or modified packets as they no morematch the definition of TC1. This problem is defined as inaccurate classification in thisthesis, and also known as traffic attribution problem in security and forensics context.


As packets escape the respective traffic class, the expected treatment by a dependentservice chain —or any other network function for that matter— is rendered incomplete.

2.1.3 State Taxonomy

Simply put, state is a collection of programmatic constructs such as file descriptors andobjects in the memory. A network function’s relation to state can be described from twoaspects. One is the scope of the state —the range of entities that it applies to— and theother is how the state is being accessed. OpenNF classifies state based on its scope asits focus is state migration [13], and StatelessNF classifies based on access pattern [15]as it investigates central state persistence.

The three forms of state with respect to its scope are per-flow, multi-flow and global.The scope of a state is said to be per-flow if it is bound to a particular flow; and multi-flow if the owning function logic spans across flows. It is common for a networkfunction to maintain constructs such as counters, which are categorized as global sate.Moreover, some state is maintained throughout or until a certain condition is met.Another maintenance concern is whether to return the allocated state back to a sourceor just dispose it when the maintenance window is over.

The access pattern signifies the level of granularity at which the processing pipelineaccesses the state for a given scope, and does what with it. A network function can createa state of which the scope is per-flow, but may access it every time a correspondingpacket arrives, or only at the start and end of the flow. These accesses could be in theform of a trivial read or a complex update. Furthermore, some state is pre-generatedwaiting to be associated with a flow, whereas some state is post-generated based oncertain fields of the packet.

A grand definitive taxonomy of state is outside the scope of this thesis. Howeverin the pursuit of breaking away from flow affinity, understanding the state space is ofprofound importance. There does not always exist a one-to-one relationship betweena network function and the state. It is often the case that a particular policy that anetwork function actuates shapes the resultant state. Consequently, certain class ofstate —a particular scope with a particular access pattern— could be mobilized in aspecific environment of execution, obviating flow affinity for select functions or policies.Section 3.5 demonstrates this for a trivial case.


2.1.4 SummaryThe bottom line is, the autonomous internetworking model with middlebox inhabitantsleaves operators with management complexity and inefficient resource utilizationproblems at hand. Automation is nothing but a computer aid for managing verticalcoordination, which becomes more and more complex with every step towards improvingresource utilization. In addition, operators’ efforts at efficiently accommodatingmiddleboxes result in excessive horizontal cooperation that begs the need for moreexpensive equipment which may bring the network to a halt otherwise.

NFV brings tremendous convenience to operators in terms of resource utilizati-onhowever forwarding ambiguity, flow affinity and inaccurate classification still remainas the key challenges to flexibility and scalability. This thesis aims to address thosethree challenges in a single framework, however considers middlebox presence inautonomous internetworking model as a dead end for achieving it. The following sectiondiscusses the reasons why, explores some alternative and clean slate approaches as wellas state-of-the-art frameworks that dissect and attempt addressing these challenges.


2.2 Programmable Networks

Management complexity, listed among the problems caused by middlebox nature in theprevious section, is not only relevant to middleboxes but also routers and switches.The autonomous nature of network nodes inevitably paves the way for complexity.Autonomy in this context can be described as the ability of decision making and takingactions by self, depending on a given configuration and information exchange. Thedecision making takes place in the control plane of an autonomous node, to which theconfiguration is passed through a proprietary interface. This configuration interfacevaries between vendors and changes over time giving cause for long term vendorspecialization commitment, which administrators have to oblige. Human error takingthe lead by far in the most common cause of autonomous network failures proves that themanagement is too complex to be trusted to human. Yet, vendor specialization rendersthe number of required personnel to manage the network directly proportional to thenumber of nodes and the degree of vendor heterogeneity [16]. Automation explainedin Section 2.1 alleviates the problem, however to a certain extent and remains as aworkaround at best.

Another drawback of autonomy lies within the horizontal cooperation. As beingindividual decision takers, autonomous nodes cooperate for the desired network-widebehavior in order to enforce enterprise-wide policy goals end-to-end. Cooperation takesplace through conventional protocols which allow the control planes of the nodes toachieve a distributed consensus on the overall network state. These protocols typicallyundergo years of standardization process and tedious interoperability tests, hinderingthe pace of research with rapidly growing traffic and evolving user-driven computationdemands from nodes within the network (middleboxes), as well as technology transferfrom academia to the industry [17]. Consequently, the autonomous approach in whichan ossifying control plane is tightly coupled with the data plane, became questionablethough radical approaches had to be taken before autonomy could cease to be a taboo.

2.2.1 Active Networking

Several of those clean slate approaches, collectively referred to as active networking,was the initial out of the box attempt at the control plane to lower the barrier toinnovation and bring research up to pace. These networks are active in the sense thatnodes’ packet processing pipeline is not bound by the conventional protocols. Thecontrol plane is rather a computation model where custom instructions can be runallowing the node to perform computation and modification on a more extensive spanof fields in a packet compared to traditional packet networks [17, 1].

2.2. PROGRAMMABLE NETWORKS 19

The organization of the three major layers of components in the active nodearchitecture is shown in Figure 2.4. The Node operating system (OS) at the bottomcan be considered as an abstraction layer between the hardware and the executionenvironments (EE) running atop. It encapsulates the implementation specific detailsof which several are contention management and provisioning hardware resources suchas computing, storage and transmission to the EEs in an isolated manner.

Figure 2.4: Active Node components (Adapted from Figure 1 of [1])

An EE is an isolated virtual machine defined by the underlying NodeOS for packetprocessing. Packets received at NodeOS on a physical link are classified then directedto the respective channel attached to an EE. An EE can be defined as simple as anIP forwarder performing common primitive operations on packets very much like atraditional router, whereas another can be a Turing complete virtual machine capable ofinterpreting instructions such as Java bytecode.

A program developed for the purpose of enacting a behavior on the traffic whichis desired by a user from the active network that the packets traverse, is referredto as an active application (AA). The two major approaches to active networking isdistinguished by how an AA comes into existence at the control plane of an active node.In the discrete approach, an AA comes either pre-deployed or on-demand on an activenode, from a code server such as in DAN [18] or from an upstream neighboring nodelike in ANTS [19] in an out-of-band fashion. Each AA has an identifier and the userencodes a reference to one or more of those identifiers in a set of headers on the packets


to invoke the desired AA(s) to process those packets at an active node they traverse.In the integrated approach, the packets are active packets in the sense that an AA iscarried within, along with the data. The embedded AA is then extracted from the activepacket and executed at an intermediary active node or at the destination host of thepacket. Active IP Option project [20] is an integrated approach where AAs are carriedin an active packet form called capsule, which upon execution, may leave informationbehind in the node that the AA fragments carried in subsequent capsules may dependon. Whereas in Smart packets project [21], an AA is designed to be completely self-contained, thereby alleviating the need for persisting information to the active node thatit is executed at.

It is common for an AA —or for a network function for that matter— to generateinformation that is essential for enacting the behavior expected from it on the traffic.This information, namely the state, is typically generated during packet processing andthe packets of the same or a correlated flow, or external factors may trigger a change init. Considering its intricate nature and tight coupling with a network function examinedin the earlier section; whether to persist the state to the node or not is the fundamentalresearch question of this thesis work, for which the active networking has become theenabler mindset.

2.2.2 Participatory NetworkingThe programmability that the active networking introduces does not only govern theintermediary nodes, but also the end hosts. An end host capable of executing andencoding arbitrary information in packets entails an end host to influence the processingof its packets by the network. Participatory networking leverages this perspective suchthat the end hosts participate in packet processing throughout the network. In mostcases, the common network stack of the end host becomes insufficient and has to beextended. One example for such extension is in ANTS where arbitrary new protocolscan be deployed to the end hosts via mobile code distribution techniques.

Despite the fact that end hosts participating in packet processing dates back to activenetworking, the term participatory networking was coined in a research in 2012 byFerguson et al., emphasizing the efforts concentrated primarily on the involvement ofthe end hosts. Throughout this thesis work however, the term participatory networkingrefers to the notion of end host’s involvement in packet processing and is not restrictedto the research in which the term was first defined.

The participation can be in several forms. PANE [22] is an extensive implementationthat provides an application programming interface (API) to the end hosts for participatingin packet processing in the network. Through this API, an end host can query the


controller for learning available path bandwidth towards a destination, place requestsfor resources such as bandwidth and security or hint the controller about the currentand predicted traffic utilization, allowing the controller to make informed decisions forimproving the services network-wide.

The particularly interesting aspect of participatory networking for this research isthe end host’s possible participation in the inner operations of a network function.ANANTA [23] is a cloud scale layer-4 load balancer where some of the inner operationsof a NAT network function is maintained at the end hosts. It leverages the fact thathypervisors are capable of scalable network processing, so it introduces a host agent(HA) component which can intercept and process packets to/from the VMs.

In order to have a basic understanding of how an end host can participate in anetwork function, briefly explaining an outbound NAT example in ANANTA shownin Figure 2.5 is helpful. The figure and the example is simplified (tunneling detailsremoved), in order to focus on the end host participation.

Figure 2.5: Operation of outbound NAT function in ANANTA

The desired behavior is simple; forward packets originating from the IP address ofthe VM to the router and forward the return traffic to a multiplexer which performs loadbalancing. In step 1, the VM initiates a flow with the source IP DIP1 and a source portportV M, (DIP,portV M). Step 2, the host agent that resides in the network stack of thehypervisor intercepts and buffers the traffic, then requests a virtual IP (VIP) and a portmapping from the ANANTA manager (AM). Step 3, AM selects an available VIP andport tuple (VIP,portAM) and installs the mapping (V IP, portAM)→ (DIP, portV M) intoevery multiplexer, then informs the HA in step 4 that (VIP,portAM) is now allocated toit. Step 5, the HA changes the source IP and port of the packets from (DIP,portV M)


to (V IP, portAM), persists this state and forwards the packets directly to the router. Instep 6, the return traffic ends up in one of the multiplexers advertising the VIP due tothe installed configuration in step 3 then it forwards the traffic to the HA accordingto the load balancing algorithm and mapping installed. Lastly, the HA translates thedestination IP and the port of the packets back to (DIP,portV M) then sends the packetsto the VM.

ANANTA demonstrates how the end host can undertake a portion of NAT networkfunction and persist state in order to realize a distributed load balancer network function.This example shows how participatory networking model can be leveraged to approachnetwork functions from a new perspective; realizing a network function as an orchestra(NFaaO) as this thesis calls it, rather than as an entity which remains to be the underlyingcause of the limitations.

2.2.3 Software Defined NetworkingActive networking paved the way for researchers to explore the control plane byintroducing programmability, which eventually led to the realization that node autonomyunderlies the most limitations. The OPENSIG has become the inspirational movementof the research and industrial community for the efforts such as xbind, Tempest andGenesis at separating the control software from the transmission hardware [24].

Among the mainstream research, Routing Control Platform (RCP) was arguablythe first bold move in this direction stating “IP routers should be lookup-and-forwardswitches, forwarding packets as rapidly as possible without being concerned aboutpath selection” [25]. Despite it had a narrow focus on routing, specifically BGP,it foreshadowed a broader view of the matter which was later embodied in the 4Dapproach. This clean slate architecture happened to be the first formalizing the problemfrom a distributed systems point of view and put the node autonomy in the cross hairs.Advocating an “extreme design point” as the authors call it, 4D architecture separatesthe decision logic from the network elements and places it into centralized servers whichhave direct control over the node resources and the forwarding state that the dataplanemaintains [26].

Ethane was among the first implementations that followed the lead of the 4Darchitecture, where the switches consisted of a simple flow table and a channel to thecontroller; forwarding packets based on the flow table that the central controller directs[27]. Depending on the policy file, the controller may install forwarding rules into thedataplane that may enact a firewall, rate-limiter or even NAT function behavior on thematched packets. Ultimately the dataplane is merely a match-action machine and it isthe control plane that shapes the hardware to its given role. If centralized, the hardware


could become a virtual local area network (VLAN) switch, a firewall or a NAT. Tounlock the innovation in this direction, an open and standardized protocol between thecontroller and the switches was a necessity.

Inspired from Ethane, OpenFlow [28] introduced an open protocol between thecontroller and the switch standardizing the interactions such as manipulating the flowtable or fetching statistics and became the cornerstone of software defined networking.While it is controversial whether or not SDN is orthogonal to active networking,the motivation was the same; “lower the entry for new ideas, increasing the rate ofinnovation in the network infrastructure” [28]. Such that even the paper’s topic was“enabling innovation in campus networks” and the term SDN was later coined in anarticle by MIT in 2009 referring to the OpenFlow paper.

Embodying the 4D architecture, SDN gives birth to a new internetworking modeldistinguished in the scope of this thesis as heteronomous internetworking model,depicted in Figure 2.6. Unlike an autonomous node, a heteronomous node (h-nodefor short) lacks an in-house control plane and assumes the controller as its controlplane instead. The agent is responsible for the secure communication through a controllink/path with the controller on its southbound API. The dataplane processes packetsbased on the rules in the flow table; if no matching rule found based on the fields of thepacket, it is sent to the controller which may typically be the case for the first packetof a new flow. The time it takes for this packet to be processed by the controller islonger than a processing that occurs in the dataplane, therefore it is also referred to asslow-path processing. It is the control applications that render the decision of how thepackets need to be handled and communicates the decision to the controller throughthe northbound API. The controller then installs the respective flow rules into the flowtables of the heteronomous nodes that lay in the path of this flow. Hence, the subsequentpackets get processed in the dataplane, also referred to as fast-path processing. Sameterms are also used for autonomous nodes except slow-path processing refers to packetdelivery to the route processor in the same chassis instead of delivery to a controllerover a network.

Active networking promoted the programmability of the control plane and SDNextended the notion by centralizing a programmable control plane as it lost its credibilityfor staying in the node once it became programmable and standardized with anopen framework. OpenFlow became the prominent protocol for implementing aheteronomous internetworking model and switch vendors introduced OpenFlow supportfor some models which became the ideal h-node. Network operators as well asresearchers now can develop their proprietary control applications for enacting thebehavior defined by policies over the network or test out new approaches. From amiddlebox and NFV standpoint, SDN brought new opportunities along with a whole


Figure 2.6: Heteronomous internetworking model

new problem domain.

2.2.4 NFV and SDN InterplayAs of version 1.5.1, an OpenFlow switch can match packets on forty-five fields andcan perform seventeen actions. Once directly available to the protocols running inautonomous nodes only, this variety of match-action is now directly at operators’disposal via control applications of their own. This makes the heteronomous internetworkingmodel the ideal candidate for more granular traffic engineering with less complexity andtakes traffic steering to a whole new level. A custom control application developed bythe operator would allow VNF presence in concert with traffic steering and overcomemany impracticalities with respect to middleboxes. This interplay also appealedresearchers to tackle the challenges inherent to the stateful nature of middleboxesdiscussed in 2.1.2, within the heteronomous internetworking model. This subsectionpresents state-of-the-art work that are closely related to this thesis.

Depicted in figure 2.7 are OpenNF [13] and Stateless Network Functions [15]; twoprominent state management frameworks that aim to improve resource utilization bymeans of efficiently scaling VNFs with specific focus on state persistence.

Figure 2.7a exhibits the flow of events during a state migration that prevents stateinconsistency without losing any packet. The initial forwarding state at N1 is shown


(a) OpenNF (b) Stateless NF

Figure 2.7: State Management Frameworks

in step one, where packets of flows one and two are configured to be forwarded tothe VNFA residing in the hypervisor on the left. Consequently, VNFA creates andmaintains the state for these two flows. In step two, the OpenNF controller takes adecision to offload flow one from VNFA to VNFB hosted in the hypervisor on theright for scaling. In step three, the OpenNF controller instructs VNFA to forward allincoming packets of flow one to the OpenNF controller to be buffered, and not processlocally. In step four, the OpenNF controller migrates the flow one’s state from VNFAto VNFB by getting then removing the state from VNFA and putting it into VNFB.Next the OpenNF controller flushes the buffered packets to VNFB in step five. Uponthe OpenNF controller’s request, the SDN controller inserts a flow rule into N1 to steerthe packets of flow one to the VNFB on the right in step six as the final step of migration.

There are two cornerstones in this framework for maintaining flow affinity. Firstis on step three; the packets of flow one, of which the state is being migrated, arenot processed at VNFA but sent to the OpenNF controller to be buffered; preventingan inconsistent treatment incident described in the scale out scenario in Figure 2.2a.Second, it is highly likely that there are still packets in transit from N1 to VNFAfor a brief moment after inserting the flow for forwarding packets to VNFB. As theinstructions passed to VNFA in step three still remain, these left out packets in-transitare forwarded to the controller, which then are immediately flushed to VNFB as theyarrive. In the end, each and every packet of the flow gets treated by the VNF thatmaintains the one and only state, even though the handling of in-transit packets wouldcause packets to arrive at VNFB out-of-order.

A general overview of the Stateless NF infrastructure is shown in Figure 2.7b. Thebasic idea is each VNF maintains state at a remote data store instead of local memory of


the instance. As the packet processing of a typical network function is tightly coupledto the state, each hypervisor that hosts VNFs is connected to the data store over a high-speed link such as Infiniband. As a result, any VNF will have access to the one andonly state for a particular network function as soon as it is instantiated. Therefore anoperator can steer the traffic to the new instance as soon as it is available for scale out,and can steer traffic from an instance scheduled to be terminated for scale in beforethe flow terminates gracefully, without worrying about flow affinity violation and itsconsequences.

Both frameworks require modifications in the VNF code. In OpenNF, VNFs haveto implement an API to expose the state they hold to the controller for realizing exportand import functions. In StatelessNF, memory access —a mere subroutine to reachthe memory allocated for state in traditional VNFs— for state lookups and updateswithin a packet processing pipeline, is replaced with remote datastore client interfaceinvocation. This means the framework becomes a native part of a VNF’s routine packetprocessing in comparison to the OpenNF where the framework is invoked only duringstate migration. As a result, the VNF’s performance is bounded by limitations inherentto the remote datastore infrastructure, as well as by the state access pattern of thenetwork function that operates in the VNF instance*. OpenNF on the other hand, incursextra latency only during the state transfer operations, which can further be reduced ifthe transfer is done peer-to-peer (P2P) bypassing the controller [29].

OpenNF offers a mechanism for orderly delivery of packets during state migration atthe expense of extra delay —in addition to the loss-free mechanism described above—as the out-of-order arrival of packets may result in unexpected behavior in some networkfunctions. In StatelessNF, the arrival of in-transit packets to the old instance and thearrival of steered packets to the new instance would result in simultaneous access to thestate by the two VNFs for a brief moment, which is not supported. Instead, StatelessNFleaves handling out-of-order packets —a likely consequence of steering traffic at willwithout being concerned about state inconsistencies— to the re-assembly capabilitiesof VNFs themselves. On the other hand, steering at will allows an operator to failoverwithout being certain that a VNF is down, as failure detection in a HA pair/group maybecome costly as explained in 2.1.2. Plus, the state is not lost when a node fails as itresides in a remote location; thereby the flows associated with that state are maintainedin a loss free manner during a failover in StatelessNF.

To address forwarding ambiguity, the two policy enforcement frameworks FlowTags[30] and SIMPLE [31] agree that the classical five tuple based traffic classification

* A firewall would require ten read/writes to the state per flow plus one read per packet, whereas a NATwould require four read/writes per flow plus one read per packet [15].


approach alone is insufficient. Both frameworks propose to encode a tag into eachpacket’s header —which supposedly remain intact throughout the packet treatmentprocess of traditional network functions— and allow the h-nodes on the forwarding pathto match on these tags to assist traffic classification, eventually rendering forwardingunambiguous. FlowTags utilizes the six bits reserved to the differentiates services (DS)in the type-of-service (ToS) field of IPv4 header for tagging packets; similar to SIMPLEwhich also utilizes Tos or VLAN fields, MPLS labels or unused fields in the IP headerfor its ProcState tag. In SIMPLE, packets departing a VNF are tagged to signify bywhich VNF it was processed, at the directly connected h-node. If we assess the hardproblem depicted in the forwarding ambiguity example in Figure 2.3b within SIMPLEframework, the packet will have a NULL tag at its first presence at N2 then will betagged with FW at its second presence, and will have a NAT tag at its third and finalpresence at N2. This will help N2 to clear the ambiguity of forwarding such that; forpackets matching a given TC defined on the five tuple, if the tag is NULL forwardout to FW, if the tag is FW forward it out to N1 and if it is NAT forward it out to itsultimate destination as the treatment desired in this example is now complete. Whereasin FlowTags, the VNFs perform tag operations in coordination with a controller whichreactively responds to the events that VNFs and h-nodes generate on the fly. For the firstpacket of each new flow, h-nodes query the controller for a forwarding entry; and VNFsquery the controller to obtain a value to tag the processed packets with. By guidingthe VNFs on what value to tag the packets with and instructing the h-nodes where toforward packets with a given tag, the framework addresses the forwarding ambiguity ina coordinated manner.

In FlowTags’ approach to tackle incomplete treatment, any VNF first persists thepacket header values that are essential for traffic classification to a hash-map in a centralcontroller per flow, should the network function modify them. Persistence is in the formof key-value where the header values before modification are mapped to that particularflow’s flow tag as the key. A VNF first queries the controller passing the flow tagof the incoming packet as the key. Next, the VNF re-writes the packet header withthe values returned to the query from the controller, allowing the packet to enter thenetwork function’s processing pipeline in its original form; thereby preventing incorrectclassification. The final step is to query the controller for a suitable flow tag to tagthe processed packet with before forwarding out. In this query, the VNF may provideadditional contextual information —such as if the payload is a cached response froma proxy— which allows the controller to make more granular forwarding decisions.Whereas instead of relying on header fields for classifying traffic that are likely to bemodified, the SIMPLE framework relies on something that is less likely to be modifiedby network functions, the payload. The first P packets of each new flow that enterand exit a VNF are forwarded to the controller from the h-node(s) connected to that


particular VNF, for a duration of W. In order to identify if the new flow(s) exiting theVNF belongs to the same traffic class with the new flow that entered the VNF, theSIMPLE controller runs an algorithm to build ratings of similarity between the payloadsof P packets that enter and exit the VNF. New flow(s) departing the VNF with highestsimilarity rating is/are considered to be related to the incoming new flow and thus belonginto the same traffic class, for which the controller then installs appropriate forwardingrules into the concerned h-nodes for fulfilling the defined treatment.

SIMPLE’s principle of avoiding modification to VNF code comes at a price; theframework may still lead to incomplete treatment as the flow correlation process isa heuristic blacbox approach and prone to errors. The larger the W, the higher theaccuracy; however the larger the induced latency and controller overhead. FlowTagshas no such problem as it is a whitebox approach; it has visibility into VNFs’ internalprocessing therefore can extract contextual information essential for classification andstore it at the controller. But it does so by introducing additional API calls subject toslow-path processing per new flow per VNF —in addition to the slow-path processinghappening per h-node— which may result in low performance* as the amount of h-nodes along the forwarding path and the length of the service chain increase. On theother hand, contextual information provided to the controller enables dynamic servicechains; for instance a static service chain defined as IDS→PROXY can be configured totransform dynamically into IDS→ IPS→ PROXY were the IDS to fire an alert, whichis not applicable with SIMPLE framework. However the more granular the treatmentpolicies get, the more tags need to be generated and the more forwarding rules need tobe installed which may become impractical considering the limited ternary content-addressable memory (TCAM) size. SIMPLE reduces the amount of h-nodes alonga path that require forwarding rules to realize given treatment by creating an overlaynetwork using tunnels, but requires h-nodes to maintain an extra table for tunnels. Asfor tagging, both frameworks utilize header fields limited in size in the packet/frameheader outside their purpose which makes it challenging for operators that alreadyutilize those fields for their pre-defined purpose. Especially FlowTags face field sizelimitations for supporting large scale production networks as it requires globally uniqueper flow tagging per path. Finally, both policy enforcement frameworks rely on lengthyoffline computations for compiling given treatment policies on a topology graph wherethe policies and the topology are assumed to change infrequently [31], in order to buildthe respective data plane realization with optimum TCAM utilization distributing theload.

The state management frameworks described explicitly address the scalability issuewhile maintaining consistent treatment, but leave the incomplete treatment problem

* For providers dealing with high rates of new and probably short flows.


out of their scope. The policy enforcement frameworks address the incompletetreatment as well as perform near-optimal load distribution, however does so by pre-computing all applicable VNF sequences per treatment policy. This limits their abilityto tackle topology changes in a timely manner to certain cases —such as node andlink failures— where the resultant topology after change can be accounted for byan alternative sequence in the pre-computed set. Scalability induced changes wouldpotentially cascade the pruned set and require full/partial re-computation in the order ofminutes/seconds. This renders the collaboration of these state management and policyenforcement frameworks impractical for operators seeking resource efficiency throughload distribution and scalability.

Network Service Header

As demonstrated in SIMPLE and FlowTags, tagging packets is a promising approachto tackle forwarding ambiguity and inaccurate classification. Network Service Header(NSH) extends this approach by incorporating a full blown header and encapsulation,rather than squeezing in a tag with limited fields. The project is carried out by aworking group called Service Function Chaining (SFC), recently formed by the InternetEngineering Task Force (IETF). The idea is to insert NSH into packets when enteringthe network, let an independent underlay transport such as VXLAN steer the packets toNFs (referred to as service functions (SF)), allow SFs to make service chaining decisionsbased on the metadata within NSH, then decapsulate NSH packets leaving the network[32].

A control plane is responsible for maintaining every possible path that can be takenin order to realize a given service chain, referred to as service path, each uniquelyidentified by a service path identifier (SPI). An SPI is accompanied by a service index(SI), which identifies the next SF that the packet shall be delivered to on the service path.The SI is decremented once an SF finishes processing the packet. In addition to the SPIand SI —4 bytes together referred to as the path header— a 16 bytes context headerallows incorporating additional metadata for sophisticated service chain realizations,as shown in Figure 2.8. In order to be able to interact with the NSH content such asSI decrement, a node has to be NSH-aware. Nodes that are not NSH-aware can beaccompanied by an SFC Proxy, which handles NSH operations on behalf.

Figure 2.9 demonstrates SFC via NSH in action for a service chain of length two. Instep one, the control plane selects a suitable SPI among the pre-calculated set of SPIsbased on the network conditions and topology to realize the matching service policy inan optimum manner. In step two at the network ingress node called service functionclassifier (SFC), respective packets are encapsulated with an NSH that is encoded with


0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31

Ver O - TTL Length - - - - MD Ty Next Protocol0

Service Path Identifier Service Index4

Mandatory Context Header(1) - Network Platform Context8

Mandatory Context Header(2) - Network Shared Context12

Mandatory Context Header(3) - Service Platform Context16

Mandatory Context Header(4) - Service Shared Context20

Optional Variable Length Context Headers24

Figure 2.8: Network Service Header

the selected path and context headers in the previous step. In step three, NSH to underlaytransport mapping table is queried, locating the VXLAN next-hop for reaching theencoded SPI/SI and forward the packet through the VXLAN tunnel. Nodes that performforwarding via bridging between the NSH domain and the underlying transport networkis called a service function forwarder (SFF). In step four, the SFF removes the transportencapsulation and forwards the NSH encapsulated packet to SF1. The NSH-aware SF1processes the packet per defined policy, encodes additional information that can be ofuse for another SF on the chain (optional), decrements the SI and forwards the NSHencapsulated packet to the SFF. The SFF(s) forward as described in step three and thepacket arrives at SF2 in step five. SF2 processes the packet per defined policy, optionallymaking use of the information encoded by SF1, then forwards the packet to the SFF.Finally in step six at the egress SFC, the NSH encapsulation is removed and the originalpacket —of which the chained service processing is complete— leaves the NSH domainto be forwarded towards its actual destination.

Currently there are two potential drawbacks of service chaining via NSH. i) Whileallowing network operators to conveniently house multiple middlebox infrastructureoperators, transport agnostic operations introduce an additional layer of managementcomplexity. That is, the control plane has to maintain and disseminate a relationshipbetween SPI/SI to underlay transport parameters such as VXLAN tunnels [33], forwhich a prominent framework does not exist as of yet. ii) Instantiation and destructionof NF instances per varying load as well as topology changes invalidate existing SPIsand require new ones to be calculated. With increased chain length and NF instances,control plane maintenance and dissemination to underlay transport scales exponentially[34]. These drawbacks are major obstacles towards achieving an elastic treatmentinfrastructure.

2.3. SUMMARY 31

Figure 2.9: Service Chaining via NSH

2.3 SummaryAs elucidated in this section; keeping treatment consistent during scaling and completealong a chain of services while maintaining the load balanced across the infrastructure isa matter of trade-offs. This thesis advocates that it is profound for the same framework totackle the limitations that statefulness impose on scalability and middlebox modificationsimpose on completeness altogether, approaching treatment as a whole in the pursuit ofstriking an ideal balance between performance and functionality for resource efficiency.Next chapter introduces a novel framework that this thesis proposes in an attemptto address the concerns that state management and policy enforcement frameworksmutually exclude, while providing satisfactory resolutions in the problem domainswhere each framework specialize.

Chapter 3

Treatment Framework

As discussed in the previous chapter, survivable and multiplexed forwarding is thefundamental functionality provided by the Internet architecture. Hence, the datagramis designed to enclose information predominantly as to where, such as source anddestination addresses as well as auxiliary information for a survivable forwardingexperience under various circumstances. When it comes to how the forwarding shouldtake place —which coincides with our definition of treatment— the IPv4 datagram isfairly limited by design. Quality of service (QoS) is a prominent example to treatmentwhich allows packet processing beyond the ordinary routing and forwarding, howeverout of the twenty bytes in the IPv4 datagram header, only one byte of ToS field is deemedfit to enclose treatment information.

Today the operators’ packet treatment demands exceed far beyond QoS as elaboratedearlier. IPv4 is not sufficient anymore; not only address space wise but also for nativelyaccommodating information for sophisticated treatment. The Treatment Framework(TRF) fleshed out in this chapter is entirely built on IPv6 protocol.

3.1 Framework Overview

The TRF’s primary goal is to create a distinction between the where and the how offorwarding, natively on the packet itself rather than by means of an underlying overlayinfrastructure. Figure 3.1 depicts the components of the framework and how theyinterface each other and the externals.

The basic idea is to encode the metadata pertinent to treatment —maintained bycontrol applications— in its own treatment header (TRH), which gets appended to IPv6datagrams as an extension header. The portion of the infrastructure where treatment onTRH appended datagrams takes place is called the treatment domain (TD). A datagram

32

3.2. VNF/MIDDLEBOX PRESENCE 33

Figure 3.1: TRF components

gets tagged with a TRH when entering the TD at the treatment edge (TE). As shownin Figure 3.1, any h-node can assume the TE role; it can be a physical one connectedto an autonomous node peering with an AS, or a virtual one residing in a hypervisorwhich is common in datacenters. The latter form of a treatment edge is referred to as aparticipatory TE (pTE) as the hypervisor becomes a part of the treatment ecosystem inTRF. This adjective does not ascribe responsibilities that one may resemble to ANANTAcase, the function is still to push and pop TRHs.

TRH appended datagrams are forwarded and treated based on the metadata containedin the TRH within TD, which is the common practice in TRF. That is, the TRH containsinformation about all the hops the packet needs to traverse, which renders a sourcerouting like forwarding mechanics throughout a TD. At each h-VNF, the packet ismatched to the corresponding NF-local policy based on the identification encoded inthe TRH. Once the chain of nodes in the TRH are traversed, the datagram is strippedoff of its TRH at a TE on its way out the TD. The metadata can additionally consist ofvendor specific information if TRAPP interfaces vendor control applciations.

3.2 VNF/Middlebox Presence

The VNFs in a TD can carry on their function specific propriety processing in anautonomous manner, however their forwarding has to be governed heteronomously.Therefore a VNF or a middlebox is expected to either implement a southbound API

34 CHAPTER 3. TREATMENT FRAMEWORK

like OpenFlow, or be accompanied with an h-node at all times. A VNF that delegatesmaking forwarding decisions in one of these ways is referred to as an h-VNF in thisframework. The form of the network function —physical or virtual— does not reallymatter as long as its control-plane is delegated. The VNFs are typically connected to avirtual switch in the hypervisor. Similarly, physical middleboxes are connected to theirtop of the rack (ToR) switch. Both switches are ideal candidates for delegating theircontrol plane. Since it is only the control of forwarding being delegated, the frameworkdoes not require a proprietary controller of its own and works with any SDN controller.

3.3 Addressing and Routing

The destination of a datagram in a TD is either an h-VNF or a TE. Therefore eachh-VNF is identified by a twenty-four bits unique identifier (UID) which is consideredas the destination address. A control application that is responsible for treatment —referred to as TRAPP— runs on top of the SDN controller overseeing the treatmentdomain. The primary responsibility of TRAPP is addressing h-VNFs and to maintainrouting along with the topology graph of the treatment domain. Addressing an h-VNFis a matter of a flow rule; if the Next h-VNF UID field of the TRH is set to X, forwardit to port n to which the middlebox/VNF is directly connected. The placement andconnections of a middlebox/VNF in production, or hot-spares for scale-out is typicallyfixed and unlikely to change. Therefore the majority of port to middlebox mappings canbe fed offline into TRAPP. Alternatively, port up/down events from the controller canbe used for online inferral methods.

The twenty-four bit UID space can be partitioned into classes. Unlike the IP addressspace for which the classification schema is pre-defined, TRF does not standardize theUID field’s utilization. This allows operators to implement their own address spaceclassification schemas; such as the lowest n bits for identification, following m bits forsome type and the next o bits for some boundry. OSPF is an ideal protocol candidateto tailor for TRF and implement in TRAPP. Once made adaptive to custom addressspace classification, such protocol could interpret traditional aggregation schemas intotreatment context. For example, a row of racks at the end of row (EoR) switch as area nand a rack of middleboxes/hypervisors as a stub area at the ToR switch. As a result, thenumber of forwarding rules per h-node in the core of the TD can be greatly reduced.

3.4. REALIZING TREATMENT POLICIES 35

3.4 Realizing Treatment Policies

Treatment is defined and enforced by means of policies, which typically specify whattraffic shall traverse what network functions and be subject to what type of processingat each. The common constituents of a treatment policy (TP) are; a traffic class, aservice chain that defines the order of h-VNFs to be traversed, and NF-local policiesthat prescribe what each h-VNF should do with the packet. Currently the TRF relieson automation for disseminating NF-local policies to VNFs. This leaves the frameworkwith the responsibility of enforcing the traversal of particular h-VNF sequences fordesired flows, merely by appending the packets with TRH and governing routing acrossthe TD.

3.4.1 TRH Construction

Second responsibility of TRAPP is to compile treatment policies into TRH constructsand negotiate resultant NF-local policies with the automation engine. It also interfaceswith the administrator for accepting policy definitions, and optionally with vendorspecific control applications for realizing NFaaO. Ultimately, a TRH is a product ofTRAPP’s ratiocination of a TP’s traffic class and the service chain, on the TD’s nodesand links topology along with available VNF resources. As the majority of treatmentpolicies are pre-known and static, most of this computation takes place offline.

A traffic class is commonly defined by the 5-tuple, and is an ideal unique identifierfor a treatment policy, referred to as TRID. During the initialization of the TD, TRAPPinterprets the treatment policies in the form of TRID being mapped to a service chain- NF-local policies pair. Then it creates two TRHs, one for the departure and one forthe return traffic of the traffic class. Finally, TRAPP installs the forwarding rules thatpushes the TRH into packets, at respective TEs so that the packets traverse the TD witha TRH at all times.

3.4.2 The Treatment Header

IPv6 natively allows encoding optional internet-layer information in separate headers,namely the extension headers. Instead of utilizing a proprietary encapsula-tion or a shimlayer, this framework chooses to embody the treatment header in the form of an IPv6extension header. Figure 3.2 depicts the proposed layout of TRH version 1.

Next Header: IPv6 allows stacking various extension headers in a datagram. This eightbits field defined by RFC6564 is used to identify the type of the IPv6 extension headerthat is right after this TRH. The type identifier for TRH is 144. It is also used as the


0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31

Next Header Hdr Ext Len Version Options0

Next h-VNF UID Flags4

Source IPv6 Address8

Destination IPv6 Address24

Source Port Destination Port40

TRID

h-VNF1 UID Length44

Variable Length Treatment Payload.

h-VNFn UID Length.

Variable Length Treatment Payload.

NFTLVs

:........Optional Header Padding(HEL+1) x 8

Figure 3.2: Treatment Header v1 Layout

protocol field of the TRID.

Hdr Ext Len (HEL): Eight bits field defined in RFC6564 to indicate the length of theTRH in 8-octet units excluding the first eight octets. The TRH is designed to enclosevariable length information and therefore may need to be padded to (HEL+1) x 8.

Version: Four bits field to encode the version of the TRF. The framework is in its firstversion at this stage, for which the layout depicted in Figure 3.2 is employed as thetreatment header.

Options: Highest two bits represent the TD type; 00 Native and 01 Hybrid, 10 and 11are not defined. A TD is called to be native if all nodes in it are heteronomous, and calledhybrid if there exists autonomous nodes within, which is outside the scope of this thesis.

Next h-VNF UID: This is the destination address of the packet within a TD andforwarding takes place based on this field at each hop.

Flags: The first bit is used to represent the packet’s directionality with respect to theTRID, which can be used for enhanced classification. Second bit is the controller flag,an h-VNF sets it to one if it makes changes to a Type Length Value (TLV). A clone ofthe packets with controller flag raised are sent to controller at the time of TRH removalat a TE.

3.4. REALIZING TREATMENT POLICIES 37

TRID: Each treatment policy is uniquely identified by the 5-tuple of the traffic class itapplies to, which is encoded into this field in the TRH by TRAPP during initialization.h-VNFs in a TD perform matching on this field rather than the actual 5-tuple forassociating flows to respective NF-local policies.

Starting from the forty-fourth octet, treatment specific information is encoded inthe form of TLVs, each identified by the UID of an h-VNF or TE. During a TRH’sinitialization, TRAPP stacks each TLV in the order that the packet must traverse pertreatment policy’s mandate. So if the service chain is defined as FW → PROXY , theh-VNF UID field of the TLV is encoded with a firewall UID at the forty-fourth octetand with a proxy UID in the TLV at the forty-fifth octet.

h-VNF UID: This field stores the UID of the h-VNF that TRAPP created this TLVfor. In the common use-case the UID is specific, meaning that this TLV is ownedby a particular h-VNF and its payload shall be accessed by that h-VNF only. Withcustom address space classification, the access can be extended to a multitude of h-VNFs enabling NFaaO approaches as exemplified in section 3.5.

Length: Eight bits field that indicate the length of the following payload in 8-octetunits, excluding the first eight octets.

Variable Length Treatment Payload: This field serves as a medium between h-VNFsand a control application when the network function is realized as an orchestra, enablingout of the ordinary approaches to packet treatment like the one discussed in section3.5. The control application can encode information pertinent to the proprietary packetprocessing of an h-VNF, and the h-VNF can update it as necessary.

3.4.3 Forwarding and ClassificationForwarding in a treatment domain takes place based on the Next VNF UID field of aTRH. At a TE when the TRH is appended for the first time, it includes the UID ofthe first h-VNF. As explained in Section 3.4.2, the TRH contains the hops UIDs asan ordered list of TLVs, first one being equal to the current Next-hVNF UID. Each h-node in the core performs only two things; a longest prefix match on this field againstthe pre-installed forwarding rules, then forward the packet out the port stated in the rule.

When the packet finally arrives at the h-VNF after traversing the core, its dataplaneperforms the match in the same fashion, then the applying rule forwards the packetout to the directly connected VNF/middlebox. Once the packet makes it back to thedataplane from the directly connected VNF after being processed, the matching rule


calls an increment action. That is; read the current Next h-VNF UID field, iterate thelist of TLVs and locate it, fetch the UID of the following TLV and set it as the Nexth-VNF UID of the TRH. In the common case, the last TLV contains a UID of a TE,allowing the packet to be forwarded on its way out the TD.

Traffic classification is a concern at two points in the framework. First one is atraditional 5-tuple classification that happens at the TE in order to identify which flowshas to be tagged with which TRH. Next classification, happens at network functions inorder to match traffic to the NF-local policy that had been disseminated via automation.Its scope is local, configuring what needs to be done with the traffic that has a particularTRID encoded in the TRH. In layman terms, this is nothing but an arbitrary matchingon an IPv6 extension header rather than the 5-tuple of the network header.

3.5 Bring Your Own State

”Metadata pertinent to treatment” as put in Section 3.1, so far consisted of only theaddresses of the hops from various vendors that need to be traversed, as an orderedlist of TLVs with no payload in the TRH. With particular setting of address spaceclassification, classes of TLVs can be associated with respective control applications.For example an operator allocates the lowest sixteen bits for h-VNF, and the highesteight bits for vendor identification. The operator assigns 0c class to vendor Cisco, thismeans TRAPP would grant Cisco’s control app full access to the VLTPs of any TLV, ofwhich the UID matches 0x0c0000/8 during TRH generation.

Leveraging this property of TRF, Bring Your Own State (BYOS) is a feature thatmobilizes certain state along with the packet which enables vendors to realize NFaaO.A vendor can maintain shippable state in its proprietary control application, appendrelevant piece of state into the VLTP of a TLV and make it available to its line ofproducts in the TD. In concert with the flexible forwarding that TRF provides the trafficcan be distributed among the h-VNFs, rendering the treatment experience at per-packetgranularity for applicable treatment policies.

3.6 Use Case: IDS-NAT Service Chain

This section presents an example realization of a treatment policy, for which certainresultant NF-local policy can be performed on mobilized state, by means of NFaaO thatemploys BYOS.

3.6. USE CASE: IDS-NAT SERVICE CHAIN 39

3.6.1 PreconditionsBefore the flow subject to this use-case arrives at the TD, it is assumed that TRAPP hasalready converged over the infrastructure. That is;

• UID address space is classified and all the h-VNFs are addressed and segmented.

• The routing protocol is in converged state and the resultant forwarding rules areinstalled at h-nodes.

• Treatment policies are compiled over the existing inventory of h-VNFs whichmeans,

– The analysis of TP semantics and the corollary state space has beencompleted.

– An allocation schema is in place for an optimum distribution of expectedload over the available resources.

– TRHs are constructed and NF-local policies have been disseminated.

Above mentioned convergence factors is kept at a bare minimum concealing thelayers of complexities of TRAPP, to simply keep the focus on the journey of a flowacross the TD.

3.6.2 Address Space Classification SchemaIn this particular use-case, the operator’s choice of address space classification is asfollows. The lowest twelve bits are allocated for h-VNF addressing, the following fourbits for vendor, and the remaining eight bits for network function type. For instance0x01/8 for all FWs, 0x02/8 all NATs, 0x0a/8 for IDSes and so on. Then 0x011/12 forall FWs from vendor A, 0x012/12 for all FWs from vendor B and 0x0af/12 for all IDSesof some vendor n.

0x00/8 is allocated for TEs, for which zones, locations and peering is a moresignificant concern rather than the vendor. So the operator utilizes vendor bits to seethe packets that have reached the end of their treatment out the TD in an optimummanner.

3.6.3 Treatment PolicyThe requirement is to inspect the number of embryonic connection attempts departingfrom a particular source to the same destination within a period of time. If passes theinspection, the same traffic shall be subject to a many-to-one 6to4 address translation,


then be forwarded to its destination. For the return traffic, the requirement is to performthe 4to6 translation, then forward the packets to its destination. The verdict of TRAPPfor such treatment is; lock the departing traffic to the same IDS as the resultant state foranalyzing distributed denial-of-service (DDoS) attack patterns is immobilizable. As forthe address translation, enable BYOS for both directions of the traffic.

To meet this requirement, T Pn is defined as the treatment of a traffic class TCnthat represents the traffic between 2001:db8:001a::/48 and 192.0.2.5/32. The departingtraffic of TCn —from 2001:db8:001a::/48 to 192.0.2.5/32— shall traverse a servicechain SCn defined as IPS→ NAT and a SC′n defined as NAT for the return traffic.

3.6.4 The Treatment ProcessThis section elaborates the execution of previously defined T Pn in three subsections.How a new flow is bonded to its TP by means of TRH appending rule installation atthe edges is explained first. After that, how TRH appended packets make their waytowards intended h-VNFs and get processed asymmetrically in the TD is described.Last subsection covers how centrally allocated state is freed upon flow termination.

3.6.4.1 Flow Setup

In step one, a packet arrives at TE1 in Figure 3.3 for the first time which gets forwardedto the controller. In step two, TRAPP classifies the packet as TCn and locates twoTRHs; T RHn for the departing traffic and T RH ′n for the return traffic. In step three,TRAPP contacts vendor A’s and B’s control applications as there exists TLVs for NATh-VNFs in T RHn and T RH ′n respectively, requiring mobilized state. In step four, bothvendor applications respond with payloads that TRAPP shall encode into the respectiveTLVs. Vendor A’s payload contains a port number picked from its pool and an IPv4address whereas the payload of vendor B contains the source IPv6 address and port ofthe incoming packet.

Finally in step five, TRAPP installs a rule to append T RHn at TE1, where the packethas initially arrived. As TE2’s UID is encoded in the last TLV, TRAPP installs a rule atTE2 to append T RH ′n for the return traffic of this flow.

3.6.4.2 Flow Treatment

Now the datagram contains T RHn and the UID in the Next h-VNF UID field is of aspecific IDS. Forwarding towards the IDS takes place in the core as explained in Section3.4.3. The datagram in Figure arrives at IDS in step one. The IDS reads the TRID fromT RHn, locates the corresponding local policy and takes the inscribed actions, creating alocal state for the flow. In step two the DDoS recognition job is done and the packet is

3.6. USE CASE: IDS-NAT SERVICE CHAIN 41

Figure 3.3: Setting up the flow

leaving the IDS where its dataplane updates the Next h-VNF UID. The highest twelvebits of the new value is addressed as “NATs of vendor A”, whereas the lowest twelvebits are set to zeroes. So in step three, the LPM process in the core forwards packetstowards the h-node that contains rules for the longest prefixes. Being a ToR directlyconnected to a farm of the NAT h-VNFs of vendor A, the h-node performs per-packetload balancing.

In step four, the packet is at one of the NAT h-VNFs of vendor A where thecorresponding local policy is being retrieved per its TRID. As instructed by the localpolicy, the NAT performs a 6to4 translation, setting the IPv4 address and the portencoded in the VLTP as the new source fields. Departing the NAT, its dataplane updatesthe Next h-VNF UID field to the final TLV’s UID, which is TE2. The datagram isforwarded towards TE2 and T RHn gets stripped leaving the TD in step five.

The flow of events are the same for the return traffic, except its packets containT RH ′n. Therefore the packet first gets forwarded to a NAT h-VNF of vendor B where4to6 translation occurs, then straight towards TE1 and leaves the TD.

3.6.4.3 Flow Termination

Certain resources in state, such as the port number in this use-case, are finite and hasto be returned back to its source once the flow terminates. This could be performedeither implicitly at TEs; when a flow expires TRAPP is notified of the expiry andinstructs vendor applications that has been involved in provisioning mobilized state.


Figure 3.4: Packet treatment

In the explicit approach, the h-VNF can zeroize the VLTP upon observing RST or FIN,set the controller flag to one and release the packet. The TE will send the stripped TRHof the departing packet to the controller due to the raised flag, and TRAPP will interpretthe zeroized payload as flow termination and inform parties accordingly.

Chapter 4

Prototype Implementation

Open Virtual Switch (OVS) and OpenFlow are the pivotal instruments that constitutethe underlying heteronomous mechanics of the framework. The majority of theimplementation took place in the form of modifications in OVS and RYU SDNcontroller for introducing the required functionalities, with OpenFlow support. Rest ofthe implementation is carried out in FastClick [35] —a performance enhanced version ofClick Modular Router [36] that supports batching, advanced multi-processing and DataPlane Development Kit (DPDK)— for building VNFs and the performance analysisenvironment. The entire implementation is designed to be in the userspace, DPDKbeing the enabling technology for fast packet processing.

4.1 New OpenFlow Match and Actions in RYU andOVS

Following are the functionalities introduced into OVS that are essential for realizing thisframework.

push trh: This OpenFlow action accepts all fields of a treatment header as arguments,including the TLVs. This is the action that TRAPP specifies in the OF rule installed at aTE to tag incoming IPv6 packets with the TRH it calculated. The action first copies thenext-protocol field of the IPv6 header into the next-header field of the TRH. Next, it setsthe next-protocol field of the IPv6 header to 144, then finally pushes the TRH betweenthe L3 and L4 headers.

pop trh: This OpenFlow action removes a TRH from the packet, setting the next-protocol of the IPv6 header to what is stored in TRH’s next-protocol field. If thecontroller flag is set in the flags field —an indication of an update in the TLV payloadfor which the TRAPP shall be informed— this action sends the TRH to the controller.

43

44 CHAPTER 4. PROTOTYPE IMPLEMENTATION

set trh nextuid: This OpenFlow action first reads the Next h-VNF UID field of theTRH, iterates the TLVs in the TRH and locates the TLV that has this UID. Then, it setsthe Next h-VNF UID of the TRH to the h-VNF UID of the following TLV. This actionis commonly used for packets leaving an h-VNF, updating its destination to the nextnode.

trh nextuid: This is an OF match that performs a longest prefix match (LPM) amongthe installed OF rules based on the Next h-VNF UID field of a packet’s TRH. The switchthen performs the forwarding action defined by the matching rule, outputting the packetto the specified port.

Installing a trh nextuid match rule requires the explicit declaration of ipv6 condition,which then implicitly assumes the existence of a treatment header, i.e. the next-headerfield of the IPv6 header is equal to 144, the protocol number for the framework. Inaddition, the OF actions mentioned above may not respect the best practices of extensionheader handling defined in RFCs 7045, in the presence of a chain of extension headers.

4.2 Controller Applications

Three controller applications ... three use cases defined in Section 5.2.1 ,leveragingthe modifications introduced into Ryu. Each application implement the followingfour components and Figure 4.1 depicts how each lie in the critical path for packetprocessing.

cache: Its purpose is to only allow the very first packet of each flow invoke thechainLogic() function. It stores a TRID to (OFPAct, packetBuf) tuple. The find()function accepts a TRID as an argument and returns an (OFPAct, packetBuf) tuple;if the OFPAct is null, the packet is appended to its packetBuf array. The add() functioninserts a TRID to OFPAct mapping to the dictionary. A flowMod() invoked by a cachehit executes packetOut until the packetBuf is depleted.

polDB: An object that stores TRID to service chain policy mappings in a dictionary.The policy consists of a list of port number tuples for the inout case, a list of VLANtuples for the VXLAN case, and a byte array packed with a TRH struct for the TRFcase. The find() function accepts TRID as an argument and returns a policy.

chainLogic: The service chaining logic is implemented in this function. For thecomparative analysis performed in Chapter 5, three service chaining logics are implementedin their respective control application; InOut, VXLAN and TRF.

4.2. CONTROLLER APPLICATIONS 45

The InOut and VXLAN reactively installs the forwarding rules for the entire chainpath, starting from the last switch in the chain all the way up to the ingress switch. Thisapproach prevents a flow table miss in the core switches. In the TRF case, forwardingrule installation in all switches except the ingress switch —the TE— is proactive bynature.

Figure 4.1: TRAPP Flowchart

Once a packet arrives at the packet in handler of a controller application, the veryfirst action is to pass it to the getTrid() function to construct the TRID, which is the 5-tuple. The cache is queried using this TRID to locate if there already exists an OFPActand if yes go ahead and install it. If not, query the polDB with the TRID to retrieve themapped steering policy, while caching this and subsequent packets that arrive with thesame 5-tuple. The policy is then passed to the chainLogic function which constructs thecorresponding OFPAct for the ingress switch. If the chaining logic is fully reactive, thisfunction first installs to the corresponding flow rules into respective switches and onlythen returns the OFPAct for the ingress switch. Last step is to add the TRID OFPActmapping into the cache, eventually followed by a FlowMod that installs the rule and

46 CHAPTER 4. PROTOTYPE IMPLEMENTATION

empties the respective packet cache with PacketOut.

4.3 Packet AnalyzerThis component is built using the existing elements in FastClick. The analyzer hastwo network interfaces; one is used of sending out the generated packets, referred to assource (SRC), and the other is for collecting them, referred to as sink SNK.

The (SRC) pipeline first generates random payloads of desired size, at desired countand rate. Then the payload is encapsulated in UDP, IPV6 and Ethernet headers in therespective order. Next element in the SRC pipeline places a number in the payloadwhich increases by one per packet, storing the number in a table for which the value isthe time. Optionally, the generated packets can be prefetched in the memory, and thenreleased for time-stamping for finer grained results. The packet then released out theinterface attached to the SRC pipeline. After traversing the attached links and nodes,the packet arrives at the interface attached to the SNK pipeline. The number in thepayload is read from the pre-defined offset, then the corresponding timestamp in thetable is compared to the current time to calculate the latency.

In order to generate unique flows per time unit, a new element handler whichincrements the source IPv6 address has been introduced into the UDPIPV6 encapsulationelement. A Script element, which runs as a separate thread from the entire pipeline, callsthis handler and increments the source IPv6 address by one at desired time intervalsin a loop. Due to the precision constraint of one millisecond of the Script element,the handler can be called maximum one thousand times per second, which is also themaximum flow generation rate of the analyzer.

Chapter 5

Analysis

This chapter presents the findings of the quantitative analysis performed on a limited-scope implementation of TRF. It starts with a detailed description of the hardware andsoftware used, followed by a brief overview of how the testbed is set up. The testcases are outlined next, elaborating the metrics that are interesting for this research andthe methods for measuring them. The results are presented and the observations arediscussed in the end, concluding the analysis chapter.

5.1 The Analysis Environment

The server on which the implementation is deployed and the analysis performed is aSupermicro X9DRT-HF+, BIOS release date 06/22/2015. It has with two E5 2800MhzCPUs and 32GB of RAM. Each CPU has ten cores which are allocated into theirrespective NUMA nodes 0 and 1 with 16GB of RAM each. Hyper Threading and allpower saving features are disabled. Ubuntu 17.10 artful is deployed as the OS withkernel 4.13.0-37, in which twenty-four huge-pages of size 1GB are allocated and theCPU cores 1-19 are isolated.

OVS version 2.9.90 along with OpenFlow13 is utilized as the h-nodes, operatingalongside with Ryu controller at version 4.29. FastClick build is at eb4ccfb and theDocker version 17.12.0-ce build is at c97c6d6. DPDK version 17.11 was used foruserspace packet processing in OVS. At the time of the analysis, DPDK implementationand OVS interoperability in FastClick is still under development.

TestbedThe testbed building objective is to represent an actual operator network as precise aspossible. The ideal testbed would consist of multiple bare-metal servers connected to

47

48 CHAPTER 5. ANALYSIS

each other with NICs that support DPDK. Alternatively, the testbed could consist of twobare-metal servers where the packets are ping-ponged between n many times where n isthe number of chained switches. However only one server was available at the disposalof this research, which presents the following challenges to building the testbed.

• Shared Datapath: Starting from version 1.9.0, OVS have been employing asingle datapath approach where any number of OVS bridges share the samedatapath. Forwarding rules of all bridge instances are rendered to dataplane matchand actions on the very same datapath. Certain combination of actions requirepackets to “recirculate”, which may become problematic with increasing numberof chained OVS bridges*. While greatly increasing performance, this nature ofOVS is an obstacle to accurately representing an actual operator network on asingle server.

• Connection Medium: The prominent method for connecting OVS bridgeinstances to each other is a patch port, an OVS proprietary connection medium.It is not only useful for connecting bridge instances at userspace, but it alsoextends the metadata bus across connected bridges, preventing operations suchas miniflow† extraction to reoccur at subsequent bridges. This performanceadvantage renders OVS bridge chaining at userspace an imprecise representationof an operator network, as operators commonly have disparate switches inseparate locations.

• Resource Sharing: The ovs-vswitchd makes the resource allocation decisi-onsand the operating system kernel schedules threads across CPU cores at its owndiscretion. Added the fact that OVS employs the shared datapath approach,emulating an instance of a switch with its own processing and memory unitsbecomes a challenge. In consequence, chaining loses its significance as long asthe OVS bridges are instantiated by the same OVS services.

The analysis environment described in Section 5.1 is utilized to build the testbeddepicted in Figure 5.1, focusing on avoiding the pitfalls presented by these challengeswhile making the best use of the limited resources.

* The TRF implementation reached the maximum recirculation depth defined by OVS when more thansix OVS bridges were chained in shared datapath.

† A miniflow is used as a key in OVS for matching packets to flows/(sub)tables.

5.1. THE ANALYSIS ENVIRONMENT 49

Figure 5.1: Testbed layout

Each OVS bridge instance —as well as the packet analyzer program written inClick— are confined into their respective Docker containers. The containers are mappedto the /dev/hugepages/ folder on the host for each OVS bridge to consume a 1GBmemory as a hugepage. Moreover, each container as well as the RYU application, arepinned to separate CPU cores. The OVS bridges are chained together using jumboframes enabled Virtual Ethernet (VETH) pairs, as shown in Figure 5.1. The RYUcontroller runs directly on the server and communicates with the OVS bridges overthe default Docker bridge connected to each container, shown with dash-dot lines in thefigure. Among the three controller applications —TRF, VXLAN and InOut— only oneruns on RYU at a time; the respective application is launched based on the test caseunder evaluation.

The OVS bridges are instantiated with userspace datapath in order to support theTRF implementation described in Section 4.1. However, the fact that VETH is a kernelelement, entails packet processing from userspace to kernel and vice versa which resultsin significant performance penalty. Profiling forwarding behavior at high packet rates isnot within the scope of this research; the current setup does suffice for benchmarking.The reader is advised not to consider the presented data as a general baseline for OVSperformance outside the context of this thesis.


5.2 Experiments and Measurement Methods

The conducted experiments in this section attempt to shed light on the followingquestions,

i) Does source-routing employed by the TRF demonstrate an acceptable forwardingperformance?

ii) Does service chaining by means of TRF demonstrate reasonable control-planeperformance?

In search for the answers, the evaluation efforts in this chapter can be categorizedunder two subjects; forwarding performance and the controller performance. Theforwarding performance section explores how the throughput is affected by the forwardingmechanisms employed in the three test cases, with respect to various frame sizes andchain lengths. Since the sole focus is the forwarding ability of service chaining, theseexperiments do not involve any middleboxes or VNFs, so the term chain length refersto the number of switches participate in the steering of particular traffic.

The control-plane side of service chaining via TRF comprise two key mechan-isms; a) routing and topology convergence during link or node state changes, and b)classify traffic, lookup service chain policy and establish the end-to-end forwarding perpolicy. Since the routing and convergence is outside the scope of this thesis, the control-plane evaluation is focused around b over a converged topology connected back to backstraight.

5.2.1 Test Cases

Various ways of traffic steering for the purpose of service chaining are designated intotheir respective case definitions as following:

InOut: This is the most basic approach to steering traffic where the flows are matchedbased on the incoming switch port and forwarded out to another port. It is expected tobe the least processor intensive case with minimum end-to-end forwarding delay. Thiscase serves as the benchmark for the rest of the test cases for performance comparison.

VXLAN: VXLAN tunneling is common in operator networks for traffic steering anddeploying service chains. This test case realizes the sample SFC implementationdescribed in Chapter 6 of [37] with slight simplifications. Each switch is assignedan IPv4 address and the MAC addresses of each are hard-coded. The VXLANtunnels between each switch is pre-established and the 5-tuple match requirement in

5.2. EXPERIMENTS AND MEASUREMENT METHODS 51

the intermediary switches is removed. The switches expect the controller application toinstall the rules with the appropriate tunnel ID and VLAN match and action, in order torealize the SFC path.

TRF: The Treatment Framework is deployed in its minimal form, utilizing the entireimplementation outlined in Section 4.1. The ingress switch performs a push trh andoutputs to a port. The intermediary switches perform a match trh nextuid, then aset trh nextuid and outputs to a port. The egress switch performs a match trh nextuid,then a pop trh and outputs to analyzer’s sink port. BYOS is not enabled, therefore eachhop takes a four bytes h-VNF TLV in the TRH header.

All three test cases have the same controller application design described in Section4.2; each implementing its own service chaining logic. For all cases, the 5-tuplematching happens only once, at the ingress switch.

5.2.2 Evaluation CriteriaFollowing aspects of performance are measured and compared between different testcases of an experiment:

Throughput

Throughput is often referred to as the amount of data that can be transferred fromone process to another through network mediums and entities. A sending processtypically encapsulates the actual intended message —referred to as payload— withinprotocol headers and releases it to a lower layer in the network stack in the form ofpackets/frames. The particularly interesting aspect of throughput in this forwardingexperiment is the rate at which OVS bridge(s) can process packets and how it is affectedin given test cases. Therefore the chosen measurement unit of throughput is packets persecond (pps) instead of bits per second.

Delay

This metric represents the amount of time between the departure of a packet frompoint A and its arrival at point B. Common factors that contribute to delay are packetprocessing latency, queuing delay, serialization and medium propagation delay. If thecontrol plane is separate from the dataplane and the flow rules are installed reactively,flow setup time is another contributor to end-to-end delay. However only the formerconcerns are under evaluation in forwarding experiments, investigating the latencyinduced by the various OpenFlow match and actions involved in different test casesfor processing packet headers.


Flow Setup Time

Signifies the amount of time it takes to establish the forwarding path towards theegress switch, from the time the first packet of a flow arrives at an ingress switch.Typical flow setup lifecycle involves classifying a flow, then performing a lookup forthe corresponding forwarding policy at the controller. Flow setup in service chainingconcept can be considered complete once the appropriate flow rules are installed in therespective switches, such that the packets forwarded from source to the destination donot enter the slow-path a second time.

Controller Bottlenecks

The controller application is responsible for identifying and deploying the definedservice chain policy for an arriving flow. This requires the application to classify theflow by extracting the 5-tuple, perform a dictionary lookup for the mapped policy usingthe 5-tuple as the key, then install the corresponding forwarding rules to respectiveswitches. Points of interests for measurement throughout control-plane operations arethe bandwidth and CPU utilization of the controller, and only the latter is within thescope of this thesis.

5.2.3 Evaluation MethodsEvaluation takes place in the form of experiments where a test case is deployed on thetestbed, executing the evaluation methods and assess performance with respect to theevaluation criteria. Each experiment is iterated twenty-three times to and the first threeiteration results are considered as “warm-up” and get discarded. The aim is to producelarge enough samples for consistent and finer-grained results.

Following are the detailed descriptions of the methods executed in respectiveexperiments.

Forwarding

Both the throughput and end-to-end delay measurements are conducted one-way only,where an iteration comprises the analyzer generating a total of 1.9 million IPv6 UDPpackets at a rate of 19Kpps from the source interface, and collects them at the sinkinterface. The packets’ 5-tuple remain the same throughout the experiments, whilethe packets sizes are tried with 70 bytes, 512 bytes and 1500 bytes when measuringthroughput, but fixed at 70 bytes when measuring delay. The forwarding rules areproactively installed in all nodes and the 5-tuple match occurs only once, at the ingressswitch.

5.2. EXPERIMENTS AND MEASUREMENT METHODS 53

Control-plane

Ideally, measuring the flow setup time would involve monitoring an alert or notificationfrom a switch indicating that the flow has successfully been installed. Howeverthe OpenFlow FlowMod message used for this purpose is not acknowledged by anOpenFlow switch by design. Therefore this research employs a heuristic approach atdetermining the flow setup time, defined by the following formula.

TFlowSetup = TSlowPath−TFastPath

• TFastPath : The delay a packet experiences when forwarded through a chain ofswitches without entering slow path, where the forwarding rules have proactivelybeen installed.

• TSlowPath : The delay a packet experiences when it enters the slow path, waits atthe controller during policy lookup and reactive rule installation(s), then sent backvia OFPPacketOut to be forwarded through the fast path it triggered the setup of.

Flow setup assessment is conducted at 10, 100 and 1000 flows per second ratesfor a period of 10 seconds, with 70 bytes payload over a chain length of three. For agiven case, first the end-to-end delay is calculated using the afore mentioned method,denoted as TFastPath. Next, the traffic generation pattern is changed as described inSection 4.3 so that each packet is classified as a new flow and forwarded via the slow-path. Thus, the policy database in the controller application is pre-populated with nmany policies* where n is the total number of flows generated. The consequent delaysamples —denoted as TSlowPath— is subtracted by the TFastPath delay samples at eachquartile in order to deduce the TFlowSetup sample distribution.

As for the controller bottlenecks, the controller application’s CPU utilization isprofiled using Pyflame [38] tool. Pyflame utilizes Linux ptrace to attach to a runningprocess and take snapshots of the Python call stack without modifying the profiledapplication. Collected stack trace samples enables identifying in which functions theCPU time is spent the most. In this experiment, a total of 100K packets of size 70bytes are pushed through three chained switches at a 1000 flows per second. Duringthis period the trftrapp.py controller application is profiled at a rate of 1000 samplesper second for 100 seconds —excluding the idle time spent, collecting a total of 48062samples.

* Policy DB lookup failures have been observed in this setting, which indicates that the srcIP is beingincremented at a slightly higher rate than set. The polDB size is set to n∗1.2 to compensate for this flawof the analyzer in these experiments.


Presentation

The data produced by experiments that measure time are visualized using box andwhisker plot for presenting the sample distribution. The 5th and 95th percentiles aretaken as the minimum and maximum respectively. Rather than plotting all samples thatfall outside the whiskers, only the 100th percentile is plotted as a single outlier.

Flamegraphs [39] tool is used for visualizing the stack trace samples generated bythe Pyflame profiling tool. Functions in a Flamegraph are represented by boxes, wherethe callee function is stacked on top of the caller function along the y-axis. Along thex-axis is the width of a box, which signifies the frequency of the function’s presencein the stack trace. The wider the box, the more the presence in the samples and hencethe more CPU time occupied. Ordering on the x-axis is purely alphabetical and thecolors have no significance other than visual differentiation in the common use case. Inthis research, full function names are truncated for better visibility and only the traceinteresting for evaluation —starting from the packet in handler— is shown for brevity.

5.3. FORWARDING PERFORMANCE 55

5.3 Forwarding PerformanceThis section presents the results of the experiments that evaluate the throughput anddelay criteria of the forwarding performance for the test cases InOut, TRF and VXLANservice chaining. Figure 5.2 presents how the packet forwarding rate —shown in they-axis— is affected as the number of chained switches increase along the x-axis.

3 7 11 1518400

18600

18800

19000

Chain Length (n)

Thr

ough

put(

Pack

ets

pers

econ

d)

InOutTRF

VXLAN70B512B1500B

Figure 5.2: Packet forwarding rate

The benchmark case InOut dropped few to none packets, demonstrating a throughputthat is almost equal to the input rate of 19Kpps through all chain lengths and packetssizes. The TRF shows a performance close to the benchmark case and outperforms theVLXAN case by up to 3.34 percent, until the chain length of eleven. It is expected thatthe TRF performance meets the VXLAN performance and start performing under it atsome point as the length of the chain continues to increase. This is because each hop isan addition of minimum four bytes to the TRH, while the VXLAN header size stays thesame regardless of the chain length. Covered next in Figure 5.3 is the end-to-end delayaspect of this same experiment, for the 70 bytes packet size scenario only.


3 7 11 150

5000

10000

15000

20000

25000

Chain Length (n)

End

-to-

End

Del

ay(µ

s)

InOutTRF

VXLAN

Figure 5.3: End-to-end Delay

Up until the chain length of seven, all three cases demonstrate similar results withmedians varying between 500-900µs. While the TRF case follows close behind thebenchmark case steadily as the chain length increases, the end-to-end latency becomeserratic for the VXLAN case starting from chain length eleven. Observed heightand wide variance in delay is attributed to the kernel space tunnel encapsulation anddecapsulation operations that have to recur at each hop. In contrast, pushing andpopping IPv6 extension header are userspace operations that take place only at theingress and the egress nodes of a chain. As a result, TRF in the best case inducesone-fourteenth of the delay that VXLAN induces (median) observed at chain lengtheleven.

5.4 Controller Performance

The results presented in this section demonstrate at what cost the service chainestablishment takes place, from a control plane standpoint. Starting with the evaluationof the flow setup time for 70 byte packets, Figure 5.4 presents the time it takes toestablish the end-to-end path over three switches, with respect to various flow arrivalrates. The lower whisker samples for the InOut and TRF cases in the 1000 flowsper second scenario is lower than the one-way trip time between the switch and the

5.4. CONTROLLER PERFORMANCE 57

controller. This indicates that the packet is forwarded via fastpath and not the slow-path, which is considered as an error for this experiment.*

10 100 10000

5000

10000

15000

20000

25000

30000

35000

40000

Flows Per Second (n)

Flow

Setu

pTi

me

(µs)

InOutTRF

VXLAN

Figure 5.4: Flow Setup Time

TRF outperforms even the benchmark case, primarily because the flow ruleinstallation is by nature reactive at the ingress switch only and proactive in the coreand egress switches. In contrast, InOut and VXLAN cases are compelled to install theentire path reactively in order avoid an explosive growth of the flow tables consideringthe limited TCAM size [40].

Moreover, the Ryu controller does not support flow rule installation to multipleswitches in parallel. A packet that enters the slow path must wait until the controllerfinishes installing the flow rules to respective switches one by one. Despite the fact thatOF FlowMod is asynchronous, it is natural to expect serial rule installation to affect theflow setup time as the number of switches in the chain increases. Figure 5.5 providesmore insight on how the controller application for TRF spends its time on the CPU at aflow arrival rate of 1000 per second.

* As a result of the asynchronous nature of the analyzer implementation explained in Section 4.3, thepacket generation rate may exceed the rate at which the srcIP is incremented (i.e. flow rate).


Figure 5.5: Partial (38.85%) flamegraph of the Ryu controller.

The packet in handler is the Ryu controller function decorator at which thetrftrapp.py application registers itself to get a copy of the packets that are sent to thecontroller. With 18671 of the entire 48062 trace samples, the handling of the packets thatenter the slow path takes up a considerable majority of the CPU time. The percentagesin the following paragraphs refer to the packet in handler samples and not the overallsample count.

Up next in the stack from right to left is the getTrid function, which accounts forthe 69.75 percent of the packet handling doing 5-tuple extraction to be used for policylookup. Using the readily available parsers in Ryu, an incoming OpenFlow messageand its payload are parsed for protocols and source/destination IPv6 addresses usingstring representations. As can be observed in the trace that roots from getTrid, most ofthe CPU time is spent on string parsing, which must be avoided in the critical path of aproduction implementation.

To the left of getTrid is add flow(3.28%), which is called for installing a flow rulefor a packet that misses the cache (see Figure 4.1). The OF protocol parser(8.09%) onits left is in charge of parsing incoming OpenFlow messages. Finally at the leftmost ofthe row is send msg (14.34%) performing OF PacketOut for the buffered packets in thecache once the very first packet of the flow that missed the cache finds its policy andtriggers an add flow.

5.5. SUMMARY 59

5.5 SummaryThe findings presented in this section can be rendered as following answers to thequestions outlined in Section 5.2;

i) Source-routing is certainly a favorable approach as a means of transport for servicechaining. While the induced end-to-end delay is insignificant, the forwarding rate isaffected as the treatment header size increases with the number of hops.

ii) The Achilies’ heel of SDN based service chaining is the initial traffic classification atthe controller, which gets worse as the number of flow arrivals increases. While beingaffected from this just like the rest of the frameworks, the implications are minimized inTRF because the classification occurs only once per flow no matter how long the chainis. Moreover, proactive rule installation in the core can reduce flow setup times andcontrol-plane bandwidth utilization.

Chapter 6

Conclusions and Future Work

The Internet is the common medium that interconnects the world and middleboxesare important to service providers that must operate and maintain them. Vendors’contribution into the Internet architecture, along with their accumulated intellect andexperience in building middleboxes has established operators’ dependence on vendors.Limited control over their resources along with the market trends has rendered thismodel no longer profitable for the operators.

With the advancements in programmable networking and virtualization, SDN andNFV became the enablers for a new model that gives operators more control over theirresources and reduces vendor dependence. The openness of this model is a negativefactor for vendors; approaches leveraging the new model that suggest modification invendors’ products are certainly crossing their red line. Completely severing vendors isnot viable either as their expertise is valuable. Nevertheless, a revolution in middleboxesand packet treatment has begun and parties have to compromise.

In this triangular relationship among the vendors, operators, and the Internetarchitecture, this thesis presents an attempt at striking an ideal balance betweenarchitectural simplicity, openness, and proprietary specialization. In its native form,the Treatment Framework; (i) employs native extension headers remaining architecturefriendly to the Internet, (ii) does not alter vendor proprietary processing, and (iii)provides operators greater control potentially allowing them to achieve a profitable,innovative, and efficient ecosystem of packet treatment.

6.1 Conclusion

The framework makes two key contributions: (i) h-VNFs classify traffic using a TRIDthat never changes, which addresses the inaccurate classification problem and (ii)

60

6.1. CONCLUSION 61

forwarding takes place towards the next treating entity and not the destination endpoint,which solves the forwarding ambiguity problem and enables asymmetric service chains.Combined with the findings in the analysis section, it can be concluded that source-routing is indeed a viable option for service chaining, answering our first question inSection 1.3.

In addition to achieving complete treatment via these contributions, the BYOSextension breaks flow affinity for certain cases, thus facilitating load distribution at aper-packet granularity. The flexibility in resource utilization allows optimum policyenforcement; however it comes at the expense of reduced goodput to throughput ratiodue to the increased header overhead. To answer the second question in Section 1.3, themiddlebox state can be mobilized and the combination with source-routing presents tobe a fit candidate for ideal service chaining. However due to the state access patterns ofnetwork functions, the state can be mobilized only for certain network functions.

Strengths and WeaknessesEven in the worst address space aggregation case, the amount of flow rules per h-nodein the core would be few enough to pre-install, as the span of destinations in a TD isdefined by the number of middleboxes and not the actual flow source and destinations.As a result, slow-path processing in the core between TEs is completely eliminated,reducing flow setup times. In the case of BYOS, the flow setup time would theoreticallyincrease depending on the number of TLVs, the size of VLTPs, and response time ofvendor control application. An evaluation of BYOS scenario for definitive answers isplanned as future work.

The inevitable downside is the decreased goodput to throughput ratio due to the TRHoverhead. This weakness is particularly salient in traffic patterns with small payloadsizes. However, compared to its nearest alternative Network Service Header (NSH),unlike TRF, NSH does not address forwarding and depends on an underlying transportprotocol. In theory, combined with the overhead incurred by its carrier protocols [41],NSH’s overhead can easily exceed TRH’s. For concrete answers, defining a servicechaining case with NSH as an addition to the test cases in Section 5.2.1 and performingevaluations, is an obvious future work.

The framework does not introduce a new protocol nor a proprietary headerencapsulation, therefore does not require awareness nor modifications to the involvednodes. Nor does it re-purpose existing fields either, therefore it can co-exist with otherprotocols and mechanisms. When used with the BYOS extension, the semantics andencoding of VLTPs between a vendor’s control application and the network function iscompletely up to the vendor, hence the framework is merely a carrier.

62 CHAPTER 6. CONCLUSIONS AND FUTURE WORK

As for the BYOS extension, the framework’s footprint at the TEs along with thecontrol path bandwidth utilization, can dramatically increase under high flow churnrate. Revisiting the use case in Section 3.6, each new flow that matches TCn will triggeran individual TRH append rule. This is because each flow needs a unique state —i.e. aunique port number in this case. In order to keep the flow table of TEs at a sustainablesize, best practice would be to go closer to the source of the flows for TRH tagging asin the participatory TE scenario.

6.2 Sustainability and Ethical Aspects

While providing the operator greater control over their resources, I believe the idealframework should maintain a harmony between vendors, operators and the architecturalprincipals of the Internet. In my opinion the first step is to abandon IPv4 as it isnot sustainable anymore, and explore its extensible successor IPv6. Unfortunatelythe majority of the research out there are still trying to “squeeze” innovation into thisscant protocol as it is still predominant in operator networks. This thesis work favorssustainability over ease of adoption and therefore refrains from any initiative that maypotentially tempt the operators to still remain on IPv4.

TRF in essence is an indirection framework, granting the carrier greater control overthe destination of user traffic. In respective settings it can become a powerful tool forinterception, allowing the incumbent to selectively wiretap user traffic. A lawful andethically grounded interception guards the balance between the privacy rights of usersand national security. Intentional or unintentional failure to do so can have profoundethical implications, therefore it is paramount that safeguards against oversight andmalice are integral to any indirection framework.

6.3 Future Work

TRF assumes path selection integral to service function chaining and is not designedto operate on an underlay transport, therefore routing and topology convergence isfundamental to TRAPP. However due to the its complexity and time constraints,this component is abandoned. Consequently, TRAPP’s offline path computation andbootstrapping time, online resource optimization and incremental TRH recompilation,are left as future work.

In the grand schema of things, development of a well-defined API is anotherrequirement, in order for vendors to easily deploy their control applications and interface

6.3. FUTURE WORK 63

with TRAPP. A resource utilization optimizer is another concern, which is essential fortaking middlebox/VNF resources into account while compiling TRHs. The optimizercould interface existing automation or network function orchestration solutions.

NSH described in Section 2.2.4 is gaining momentum and its service chainingperformance analysis would be a valuable contribution. Incorporating NSH as thefourth test case into the experiments in Chapter 5, as well as conducting them at ordersof magnitude higher packet rates over chains connected at userspace, are set as theimmediate future work.

Lastly, implementation of a NAT64 prototype which was listed among the goalsin Section 1.4, is left undone. The performance comparison between a VNF withtraditional packet processing pipeline where the state is maintained locally, and theidentical VNF with a pipeline that reads state from TRH, is an interesting evaluationplanned to be carried out next.

6.3.1 Opportunities and HardshipsRecent study shows that ISPs have an inclination towards deploying carrier-grade NAT(CGN), but dragging behind in IPv6 deployment [42]. CGN presence is salient in themajority of Tier 3 and 2 ISPs; above 92% in cellular and up to 18% in the rest of theirnetworks. This is a frame of opportunity for a NAT64 implementation that leveragesBYOS, to solve their IPv4 address space depletion and accelerate their transition into itssustainable successor at the same time, and make it particularly appealing by providinga per-packet flexible treatment experience.

The framework inherently resolves the forwarding accountability problem [43]; itimpossible for an entity outside the TD to deviate, detour, or forge out of the determinedroute. Retaining forensic trails in the cloud is also a hot topic [44], for which archivingaggregates of popped TRHs could be beneficial for traceability and non-repudiation.

In a later version of the TRH, the Next h-VNF UID field size could be extended toenable more flexible aggregation schemas that could allowing inter-TD treatment. Thiswould be interesting as middlebox outsourcing is a trending phenomenon [45]. Some ofthe approaches, such as Zscaler, employs host agents, which in TRF would be equivalentto the host becoming the pTE. Although this is the most optimum of scenarios that theTRF can get, it introduces security and fragmentation challenges due to the limitedmaximum transmission unit (MTU) size at the Internet exchange points (IXP).

One of the main roadblocks to swift adoption of this framework is the ongoingambiguity of the handling of IPv6 extension headers. RFC 7045 rules out many of

64 CHAPTER 6. CONCLUSIONS AND FUTURE WORK

them (however at the time of this writing, this RFC is still in the “proposed” status).Also it does not address the case where network functions spawn new connectionson behalf of an incoming connection. In TRF, the network function must copy thetriggering datagram’s extension headers to the new flow. Otherwise the new flows cannot be attributed to the triggering flow, which resurrects the inaccurate classificationproblem. If the standard does not support copy behavior in future, then TRF wouldrequire vendors to implement it in order to solve inaccurate classification.

6.3.2 Insights and SuggestionsThe art and science of decoupling or producing shippable state is a hard problem andBYOS barely scratches the surface of it. Concluding whether or not a TP could entirelyor partially be realized in such manner requires TRAPP to have certain intelligenceabout a treatment policy’s decomposition. The relationship between a TP’s constituents:NF-local policy, its actuator, and the resultant state could be dissected via StateAlyzr[47] and a tailored Header Space Analysis [48] to master the state space and pave theway for NFaaO.

In terms of making customizations in the OVS code, the timing of this thesis (in thelight of the development plans for the OVS mainline) was perhaps a bit unfortunate.That is because in the upcoming versions, the OVS team plans to integrate the P4language [49], thus allowing experimenters to easily compile their own custom fieldsand headers along with the respective match and actions against the OVS dataplane.This could easily shrink months of work down to a couple of days.

Bibliography

[1] K. L. Calvert, S. Bhattacharjee, E. Zegura, and J. Sterbenz, “Directions in activenetworks,” IEEE Communications Magazine, vol. 36, no. 10, pp. 72–78, Oct 1998.doi: 10.1109/35.722139

[2] M. Page, L. Rossi, and C. Rand, “A Viable FutureModel for the Internet,” A.T Kearney, Dec 2010. [Online].Available: http://www.atkearney.com.tr/documents/10192/557848/Viable FutureModel for Internet.pdf/4b98dac5-0c99-4439-9292-72bfcd7a6dd1

[3] S. Schoen, “Comcast is forging packets to interfere with user traffic,” ElectronicFrontier Foundation, Oct 2007. [Online]. Available: https://www.eff.org/deeplinks/2007/10/eff-tests-agree-ap-comcast-forging-packets-to-interfere

[4] “An update on Paxfire and search redirection,” Electronic FrontierFoundation, Aug 2011. [Online]. Available: https://www.eff.org/deeplinks/2011/08/update-paxfire-and-search-redirection

[5] C. Kang, “FCC fines verizon $1.25m for blocking tethering apps,” WashingtonPost, Jul 2012. [Online]. Available: https://www.washingtonpost.com/blogs/post-tech/post/fcc-fines-verizon-125m-for-blocking-tethering-apps/2012/07/31/gJQAXjRLNX blog.html?utm term=.288ae33eba53

[6] D. Clark, “The design philosophy of the DARPA internet protocols,” inSymposium Proceedings on Communications Architectures and Protocols, ser.SIGCOMM ’88. ACM. doi: 10.1145/52324.52336. ISBN 978-0-89791-279-2pp. 106–114. [Online]. Available: http://doi.acm.org/10.1145/52324.52336

[7] J. M. McQuillan and D. C. Walden, “The ARPA network design decisions,” vol. 1,no. 5, pp. 243–289. doi: 10.1016/0376-5075(77)90014-9. [Online]. Available:http://www.sciencedirect.com/science/article/pii/0376507577900149

[8] B. Carpenter and S. Brim, “Middleboxes: Taxonomy and Issues,” InternetRequests for Comments, RFC Editor, RFC 3234, February 2002. [Online].Available: http://www.rfc-editor.org/rfc/rfc1654.txt

65

http://www.atkearney.com.tr/documents/10192/557848/Viable_Future_Model_for_Internet.pdf/4b98dac5-0c99-4439-9292-72bfcd7a6dd1

http://www.atkearney.com.tr/documents/10192/557848/Viable_Future_Model_for_Internet.pdf/4b98dac5-0c99-4439-9292-72bfcd7a6dd1

https://www.eff.org/deeplinks/2007/10/eff-tests-agree-ap-comcast-forging-packets-to-interfere

https://www.eff.org/deeplinks/2007/10/eff-tests-agree-ap-comcast-forging-packets-to-interfere

https://www.eff.org/deeplinks/2011/08/update-paxfire-and-search-redirection

https://www.eff.org/deeplinks/2011/08/update-paxfire-and-search-redirection

https://www.washingtonpost.com/blogs/post-tech/post/fcc-fines-verizon-125m-for-blocking-tethering-apps/2012/07/31/gJQAXjRLNX_blog.html?utm_term=.288ae33eba53



http://doi.acm.org/10.1145/52324.52336

http://www.sciencedirect.com/science/article/pii/0376507577900149

http://www.rfc-editor.org/rfc/rfc1654.txt

66 BIBLIOGRAPHY

[9] M. Walfish, J. Stribling, M. Krohn, H. Balakrishnan, R. Morris, and S. Shenker,“Middleboxes no longer considered harmful,” in Proceedings of the 6thConference on Symposium on Opearting Systems Design & Implementation -Volume 6, ser. OSDI’04. USENIX Association, pp. 15–15. [Online]. Available:http://dl.acm.org/citation.cfm?id=1251254.1251269

[10] HR-tail-f-NETCONF-WP-10-08-13.pdf. [Online]. Available:http://www.tail-f.com/wordpress/wp-content/uploads/2013/10/HR-Tail-f-NETCONF-WP-10-08-13.pdf

[11] IDC home: The premier global market intelligence firm. [Online]. Available:http://www.idc.com/

[12] M. Chiosi, C. Cui, H. Deng, U. Michel, H. Damker, and T. M.Ogaki, Kenichi, “Network functions virtualisation. An introduction, benefits,enablers, challenges & call for action,” p. 16. [Online]. Available: https://portal.etsi.org/nfv/nfv white paper.pdf

[13] A. Gember-Jacobson, R. Viswanathan, C. Prakash, R. Grandl, J. Khalid, S. Das,and A. Akella, “OpenNF: Enabling innovation in network function control,” inProceedings of the 2014 ACM Conference on SIGCOMM, ser. SIGCOMM ’14.ACM. doi: 10.1145/2619239.2626313. ISBN 978-1-4503-2836-4 pp. 163–174.[Online]. Available: http://doi.acm.org/10.1145/2619239.2626313

[14] A. R. Sharafat, S. Das, G. Parulkar, and N. McKeown, “MPLS-TE andMPLS VPNS with openflow,” in Proceedings of the ACM SIGCOMM 2011Conference, ser. SIGCOMM ’11. ACM. doi: 10.1145/2018436.2018516. ISBN978-1-4503-0797-0 pp. 452–453. [Online]. Available: http://doi.acm.org/10.1145/2018436.2018516

[15] M. Kablan, A. Alsudais, E. Keller, and F. Le, “Stateless network functions:Breaking the tight coupling of state and processing.” in NSDI, 2017, pp. 97–112.

[16] J. Sherry, S. Hasan, C. Scott, A. Krishnamurthy, S. Ratnasamy, and V. Sekar,“Making middleboxes someone else’s problem: Network processing as a cloudservice,” in Proceedings of the ACM SIGCOMM 2012 Conference on Applications,Technologies, Architectures, and Protocols for Computer Communication, ser.SIGCOMM ’12. ACM. doi: 10.1145/2342356.2342359. ISBN 978-1-4503-1419-0 pp. 13–24. [Online]. Available: http://doi.acm.org/10.1145/2342356.2342359

[17] N. Feamster, J. Rexford, and E. Zegura, “The road to SDN,” vol. 11,no. 12, pp. 20:20–20:40. doi: 10.1145/2559899.2560327. [Online]. Available:http://doi.acm.org/10.1145/2559899.2560327

http://dl.acm.org/citation.cfm?id=1251254.1251269

http://www.tail-f.com/wordpress/wp-content/uploads/2013/10/HR-Tail-f-NETCONF-WP-10-08-13.pdf

http://www.tail-f.com/wordpress/wp-content/uploads/2013/10/HR-Tail-f-NETCONF-WP-10-08-13.pdf

http://www.idc.com/

https://portal.etsi.org/nfv/nfv_white_paper.pdf

https://portal.etsi.org/nfv/nfv_white_paper.pdf

http://doi.acm.org/10.1145/2619239.2626313

http://doi.acm.org/10.1145/2018436.2018516

http://doi.acm.org/10.1145/2018436.2018516

http://doi.acm.org/10.1145/2342356.2342359

http://doi.acm.org/10.1145/2559899.2560327

BIBLIOGRAPHY 67

[18] D. Decasper and B. Plattner, “DAN: distributed code caching for activenetworks,” in IEEE INFOCOM ’98. Seventeenth Annual Joint Conference ofthe IEEE Computer and Communications Societies. Proceedings, vol. 2. doi:10.1109/INFCOM.1998.665081 pp. 609–616 vol.2.

[19] D. J. Wetherall, J. V. Guttag, and D. L. Tennenhouse, “ANTS: a toolkit for buildingand dynamically deploying network protocols,” in 1998 IEEE Open Architecturesand Network Programming. doi: 10.1109/OPNARC.1998.662048 pp. 117–129.

[20] D. J. Wetherall and D. L. Tennenhouse, “The ACTIVE IP option,” in Proceedingsof the 7th Workshop on ACM SIGOPS European Workshop: Systems Support forWorldwide Applications, ser. EW 7. ACM. doi: 10.1145/504450.504457 pp.33–40. [Online]. Available: http://doi.acm.org/10.1145/504450.504457

[21] B. Schwartz, A. W. Jackson, W. T. Strayer, W. Zhou, R. D. Rockwell, andC. Partridge, “Smart packets for active networks,” in 1999 IEEE SecondConference on Open Architectures and Network Programming Proceedings, 1999.OPENARCH ’99. doi: 10.1109/OPNARC.1999.758557 pp. 90–97.

[22] A. D. Ferguson, A. Guha, C. Liang, R. Fonseca, and S. Krishnamurthi,“Participatory networking: An API for application control of SDNs,”in Proceedings of the ACM SIGCOMM 2013 Conference on SIGCOMM, ser.SIGCOMM ’13. ACM. doi: 10.1145/2486001.2486003. ISBN 978-1-4503-2056-6 pp. 327–338. [Online]. Available: http://doi.acm.org/10.1145/2486001.2486003

[23] P. Patel, D. Bansal, L. Yuan, A. Murthy, A. Greenberg, D. A. Maltz, R. Kern,H. Kumar, M. Zikos, H. Wu, C. Kim, and N. Karri, “Ananta: Cloud scaleload balancing,” in Proceedings of the ACM SIGCOMM 2013 Conference onSIGCOMM, ser. SIGCOMM ’13. ACM. doi: 10.1145/2486001.2486026. ISBN978-1-4503-2056-6 pp. 207–218. [Online]. Available: http://doi.acm.org/10.1145/2486001.2486026

[24] A. T. Campbell, I. Katzela, K. Miki, and J. Vicente, “Open signaling for ATM,internet and mobile networks (OPENSIG’98),” vol. 29, no. 1, pp. 97–108. doi:10.1145/505754.505762. [Online]. Available: http://doi.acm.org/10.1145/505754.505762

[25] N. Feamster, H. Balakrishnan, J. Rexford, A. Shaikh, and J. van der Merwe, “Thecase for separating routing from routers,” in Proceedings of the ACM SIGCOMMWorkshop on Future Directions in Network Architecture, ser. FDNA ’04. ACM.doi: 10.1145/1016707.1016709. ISBN 978-1-58113-942-6 pp. 5–12. [Online].Available: http://doi.acm.org/10.1145/1016707.1016709

http://doi.acm.org/10.1145/504450.504457

http://doi.acm.org/10.1145/2486001.2486003

http://doi.acm.org/10.1145/2486001.2486026

http://doi.acm.org/10.1145/2486001.2486026

http://doi.acm.org/10.1145/505754.505762

http://doi.acm.org/10.1145/505754.505762

http://doi.acm.org/10.1145/1016707.1016709

68 BIBLIOGRAPHY

[26] A. Greenberg, G. Hjalmtysson, D. A. Maltz, A. Myers, J. Rexford, G. Xie,H. Yan, J. Zhan, and H. Zhang, “A clean slate 4d approach to network controland management,” vol. 35, no. 5, pp. 41–54. doi: 10.1145/1096536.1096541.[Online]. Available: http://doi.acm.org/10.1145/1096536.1096541

[27] M. Casado, M. J. Freedman, J. Pettit, J. Luo, N. McKeown, andS. Shenker, “Ethane: Taking control of the enterprise,” in Proceedingsof the 2007 Conference on Applications, Technologies, Architectures, andProtocols for Computer Communications, ser. SIGCOMM ’07. ACM.doi: 10.1145/1282380.1282382. ISBN 978-1-59593-713-1 pp. 1–12. [Online].Available: http://doi.acm.org/10.1145/1282380.1282382

[28] N. McKeown, T. Anderson, H. Balakrishnan, G. Parulkar, L. Peterson, J. Rexford,S. Shenker, and J. Turner, “OpenFlow: Enabling innovation in campus networks,”vol. 38, no. 2, pp. 69–74. doi: 10.1145/1355734.1355746. [Online]. Available:http://doi.acm.org/10.1145/1355734.1355746

[29] B. Kothandaraman, M. Du, and P. Skoldstrom, “Centrally controlled distributedVNF state management,” in Proceedings of the 2015 ACM SIGCOMM Workshopon Hot Topics in Middleboxes and Network Function Virtualization, ser.HotMiddlebox ’15. ACM. doi: 10.1145/2785989.2785996. ISBN 978-1-4503-3540-9 pp. 37–42. [Online]. Available: http://doi.acm.org/10.1145/2785989.2785996

[30] S. K. Fayazbakhsh, V. Sekar, M. Yu, and J. C. Mogul, “FlowTags:Enforcing network-wide policies in the presence of dynamic middleboxactions,” in Proceedings of the Second ACM SIGCOMM Workshop onHot Topics in Software Defined Networking, ser. HotSDN ’13. ACM.doi: 10.1145/2491185.2491203. ISBN 978-1-4503-2178-5 pp. 19–24. [Online].Available: http://doi.acm.org/10.1145/2491185.2491203

[31] Z. A. Qazi, C.-C. Tu, L. Chiang, R. Miao, V. Sekar, and M. Yu, “SIMPLE-fying middlebox policy enforcement using SDN,” in Proceedings of the ACMSIGCOMM 2013 Conference on SIGCOMM, ser. SIGCOMM ’13. ACM.doi: 10.1145/2486001.2486022. ISBN 978-1-4503-2056-6 pp. 27–38. [Online].Available: http://doi.acm.org/10.1145/2486001.2486022

[32] P. Quinn and J. Guichard, “Service function chaining: Creating a service plane vianetwork service headers,” vol. 47, no. 11, pp. 38–44. doi: 10.1109/MC.2014.328

[33] G. Davoli, W. Cerroni, C. Contoli, F. Foresta, and F. Callegati, “Implementationof service function chaining control plane through OpenFlow,” in 2017 IEEE

http://doi.acm.org/10.1145/1096536.1096541

http://doi.acm.org/10.1145/1282380.1282382

http://doi.acm.org/10.1145/1355734.1355746

http://doi.acm.org/10.1145/2785989.2785996

http://doi.acm.org/10.1145/2785989.2785996

http://doi.acm.org/10.1145/2491185.2491203

http://doi.acm.org/10.1145/2486001.2486022

BIBLIOGRAPHY 69

Conference on Network Function Virtualization and Software Defined Networks(NFV-SDN). doi: 10.1109/NFV-SDN.2017.8169852 pp. 1–4.

[34] S. Kulkarni, M. Arumaithurai, K. K. Ramakrishnan, and X. Fu, “Neo-NSH:Towards scalable and efficient dynamic service function chaining of elasticnetwork functions,” in 2017 20th Conference on Innovations in Clouds, Internetand Networks (ICIN). doi: 10.1109/ICIN.2017.7899429 pp. 308–312.

[35] T. Barbette, C. Soldani, and L. Mathy, “Fast userspace packet processing,”in Proceedings of the Eleventh ACM/IEEE Symposium on Architectures forNetworking and Communications Systems, ser. ANCS ’15. Washington, DC,USA: IEEE Computer Society, 2015. ISBN 978-1-4673-6632-8 pp. 5–16.[Online]. Available: http://dl.acm.org/citation.cfm?id=2772722.2772727

[36] E. Kohler, R. Morris, B. Chen, J. Jannotti, and M. F. Kaashoek, “Theclick modular router,” ACM Trans. Comput. Syst., vol. 18, no. 3, pp.263–297, Aug. 2000. doi: 10.1145/354871.354874. [Online]. Available:http://doi.acm.org/10.1145/354871.354874

[37] “L4-l7 service function chaining solution architecture,” OpenNetwork Foundation, Jun 2015. [Online]. Available: https://www.opennetworking.org/wp-content/uploads/2014/10/L4-L7 ServiceFunction Chaining Solution Architecture.pdf

[38] Pyflame: A ptracing profiler for python. [Online]. Available: https://github.com/uber/pyflame

[39] B. Gregg, “Visualizing performance with flame graphs.” Santa Clara, CA:USENIX Association, 2017.

[40] K. Lakshminarayanan, A. Rangarajan, and S. Venkatachary, “Algorithms foradvanced packet classification with ternary cams,” in Proceedings of the 2005Conference on Applications, Technologies, Architectures, and Protocols forComputer Communications, ser. SIGCOMM ’05. New York, NY, USA:ACM, 2005. doi: 10.1145/1080091.1080115. ISBN 1-59593-009-4 pp. 193–204.[Online]. Available: http://doi.acm.org/10.1145/1080091.1080115

[41] J. M. Tilli and R. Kantola, “Data plane protocols and fragmentation for 5g,”in 2017 IEEE Conference on Standards for Communications and Networking(CSCN), Sept 2017. doi: 10.1109/CSCN.2017.8088623 pp. 207–213.

[42] P. Richter, F. Wohlfart, N. Vallina-Rodriguez, M. Allman, R. Bush, A. Feldmann,C. Kreibich, N. Weaver, and V. Paxson, “A multi-perspective analysis ofcarrier-grade nat deployment,” in Proceedings of the 2016 Internet Measurement


http://doi.acm.org/10.1145/354871.354874

https://www.opennetworking.org/wp-content/uploads/2014/10/L4-L7_Service_Function_Chaining_Solution_Architecture.pdf



https://github.com/uber/pyflame

https://github.com/uber/pyflame

http://doi.acm.org/10.1145/1080091.1080115

70 BIBLIOGRAPHY

Conference, ser. IMC ’16. New York, NY, USA: ACM, 2016. doi:10.1145/2987443.2987474. ISBN 978-1-4503-4526-2 pp. 215–229. [Online].Available: http://doi.acm.org.focus.lib.kth.se/10.1145/2987443.2987474

[43] T. Sasaki, C. Pappas, T. Lee, T. Hoefler, and A. Perrig, “Sdnsec: Forwardingaccountability for the sdn data plane,” in 2016 25th International Conferenceon Computer Communication and Networks (ICCCN), Aug 2016. doi:10.1109/ICCCN.2016.7568569 pp. 1–10.

[44] Upcoming challenges in cloud forensics. [Online]. Available: https://www.researchgate.net/post/cloud forensics4

[45] C. Lan, J. Sherry, R. A. Popa, S. Ratnasamy, and Z. Liu, “Embark: Securelyoutsourcing middleboxes to the cloud,” in Proceedings of the 13th UsenixConference on Networked Systems Design and Implementation, ser. NSDI’16.Berkeley, CA, USA: USENIX Association, 2016. ISBN 978-1-931971-29-4pp. 255–273. [Online]. Available: http://dl.acm.org/citation.cfm?id=2930611.2930629

[46] W. Sun, H. Wei, Z. Ji, Q. Zhang, and C. Lin, “A collaborated IPv6-packetsmatching mechanism base on flow label in OpenFlow (analysing OVS IPv6perf),” in Collaborative Computing: Networking, Applications, and Worksharing,S. Guo, X. Liao, F. Liu, and Y. Zhu, Eds. Springer International Publishing,vol. 163, pp. 14–25. ISBN 978-3-319-28909-0 978-3-319-28910-6. [Online].Available: http://link.springer.com/10.1007/978-3-319-28910-6 2

[47] J. Khalid, A. Gember-Jacobson, R. Michael, A. Abhashkumar, and A. Akella,“Paving the way for NFV: Simplifying middlebox modifications usingStateAlyzr,” in Proceedings of the 13th Usenix Conference on NetworkedSystems Design and Implementation, ser. NSDI’16. USENIX Association. ISBN978-1-931971-29-4 pp. 239–253. [Online]. Available: http://dl.acm.org/citation.cfm?id=2930611.2930628

[48] P. Kazemian, G. Varghese, and N. McKeown, “Header space analysis: Staticchecking for networks,” in Proceedings of the 9th USENIX Conference onNetworked Systems Design and Implementation, ser. NSDI’12. Berkeley,CA, USA: USENIX Association, 2012, pp. 9–9. [Online]. Available: http://dl.acm.org/citation.cfm?id=2228298.2228311

[49] P. Bosshart, D. Daly, G. Gibb, M. Izzard, N. McKeown, J. Rexford, C. Schlesinger,D. Talayco, A. Vahdat, G. Varghese, and D. Walker, “P4: Programming protocol-independent packet processors,” SIGCOMM Comput. Commun. Rev., vol. 44,

http://doi.acm.org.focus.lib.kth.se/10.1145/2987443.2987474

https://www.researchgate.net/post/cloud_forensics4

https://www.researchgate.net/post/cloud_forensics4



http://link.springer.com/10.1007/978-3-319-28910-6_2





BIBLIOGRAPHY 71

no. 3, pp. 87–95, Jul. 2014. doi: 10.1145/2656877.2656890. [Online]. Available:http://doi.acm.org/10.1145/2656877.2656890

http://doi.acm.org/10.1145/2656877.2656890

TRITA -EECS-EX-2019:46

www.kth.se

Treatment Frameworkkth.diva-portal.org/smash/get/diva2:1299705/FULLTEXT01.pdfAbstract The middlebox architecture is long known for its inharmonious presence within the Internet architecture.

Documents