Data Dissemination using Information-Centric Networking by Ali Shariatmadari A thesis submitted in conformity with the requirements for the degree of Doctor of Philosophy Graduate Department of Electrical and Computer Engineering University of Toronto c Copyright 2016 by Ali Shariatmadari
140
Embed
Data Dissemination using Information-Centric Networking · 2.1 Information-CentricNetworking Information-Centric Networking (ICN) [2,3,13] is a clean slate networking paradigm that
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
Data Dissemination using Information-CentricNetworking
by
Ali Shariatmadari
A thesis submitted in conformity with the requirementsfor the degree of Doctor of Philosophy
Graduate Department of Electrical and Computer EngineeringUniversity of Toronto
over time. The publisher satisfies this Interest packet when it has new infor-
mation available to publish. Then the brokers will receive that information
and immediately sends the next Interest packet. The Interest packets at the
publisher may expire without any new data. The expiration of the Interest
means that if the publisher does not generate any new data for a while, it can-
not send it to the broker. However, this is not a problem, since the publisher
will re-initiate the “start” process discussed above periodically, and will receive
another Interest packet from the broker. It must be noted that the sequence
number is a choice made by the publisher.
3.1.2 Subscriber-Broker Exchange
Similar data exchange will happen on the subscriber side as denoted in Fig. 3.3.
On start, the subscriber will send its alive status to the broker. This name of
this Interest packet consists of, the name of the broker, the action /sub/start,
and the full path of the subscriber, plus the subscriber’s specific configuration.
The broker acknowledges this action and responds with a set of configurations.
Chapter 3. Data Dissemination in CVST 48
Subscriber
/<sub_id>/match/<data_name>
/broker/sub/data/<#sub_id>/<seq#>
/borker/sub/<#sub_id>/<new seq#>
/broker/sub/start/<sub_id>/<config>
ACK
Broker
/<data_name>
Data
Figure 3.3: Subscriber-Broker Communication
Then the subscriber sends an Interest packet for the data notification. The
name of this Interest consists of the name of the broker, the action /sub/data,
the path of the subscriber and the sequence number of the data the subscriber
has already received. When the broker acquires a data from the publisher that
matches what the subscriber has requested, the name of that data will be sent
to the subscriber. The subscriber will then use that name to claim the data.
After receiving of the data, the subscriber sends the next data request.
Subscribers are required to send the Interest packets periodically. This
periodic Interest acts as a heartbeat for the subscriber and provides more
flexibility for the broker. For example, the broker will be able to pause the
data matching for a subscriber if the heartbeat stops. It is also possible to
register a subscriber without the heartbeat. In that case, the subscriber will
provide a callback name in the registration process and the broker will send an
Interest similar to the start Interest of the publisher to notify the subscriber of
Chapter 3. Data Dissemination in CVST 49
the data. Then the subscriber will send the data request Interest accordingly.
3.1.3 Discussion
In this section, we discuss the reasoning behind our design and its benefits in
different scenarios.
Simple is better than complex
One of the advantages of the presented naming design is to simplify the ar-
chitecture of the broker and to minimize the necessity of keeping the state of
the system as much as possible. For example, in the publisher-broker commu-
nication, a publisher either sends out the “start” Interest to notify the broker,
or satisfies the Interests from the broker. On the other hand, the broker only
needs to have access to the latest data it has received from the publisher.
The same reasoning may be applied to the side of the subscriber. By
employing the heartbeat, the subscriber will become responsible for keeping
its status alive for the period of its interest in the data. The broker can be
configured to stop data matching for the subscriber due to lack of heartbeat
signal.
Mobility
One of the main advantages provided by the IDD layer is the mobility support.
The IDD layer uses point-to-point communication protocol, unlike TCP, which
is an end-to-end protocol. Nodes will communicate by knowing the name of the
data that they are interested in, and the network will route that data towards
Chapter 3. Data Dissemination in CVST 50
the destination. So a mobile publisher will continue to receive Interest packets
from the broker, and the broker will receive the data, without the need to
re-initiate the communication.
If the network becomes partitioned, the publisher will not receive the In-
terest packets from the broker. When new data is available, it will go through
the “start” process, since there is no pending Interest from the broker by the
publisher side. Based on the publisher configuration, this can be repeated
indefinitely. The publisher can also be configured to store the historical data
locally. When the link is back on, the broker will be notified of the existence
of the new data and will send the Interest for the data. The sequence number
in the Interest contains the latest information the broker has received. The
publisher, based on its configuration, may have stored the historical data, and
therefore, will send them to the broker, or the historical data was not needed,
and the last available data will be forwarded to the broker. For example, the
history of a live video stream will not be saved; however, the log of a traffic
sensor data will be saved, and if asked by the broker, will be sent over. The
historical data can be purged when the broker sends the new Interest packet
with a new sequence number acknowledging the receive of the data. This sce-
nario also happens if the Interest packet sent from the broker to the publisher
is dropped.
If the data packet from the publisher is dropped, the broker will not send
new Interest packet to the publisher, and there will be no Interest packet by
the publisher side for the next data. To resend the data again, the publisher
will go into the “start” process after a deadline and will notify the broker of
Chapter 3. Data Dissemination in CVST 51
the existence of the data. The deadline is configurable, and it is on the scale
of the round-trip time.
On the side of the subscriber, the heartbeat will be received by the broker
even if the subscriber is mobile. Interest packet acts as a breadcrumb for the
Data packet and data will take the reverse path of the Interest packet until it
reaches the subscriber. If the subscriber has moved from its original place, it
will resend another heartbeat interest. The heartbeat Interest will either be
satisfied by one of the upper layer routers, due to in-network caching, or by
the broker itself.
The heartbeat will also resolve the network partitioning scenario. After
network partitioning, the subscriber will not receive any new data, and the
heartbeat Interest will contain an old sequence number. After resolving of the
network partitioning, the broker will receive a heartbeat with an old sequence
number. If the broker has been configured to store the historical data for
the subscriber, the name of the historical data will be sent to the subscriber.
Otherwise, the name of the new data will be sent over. The same process
happens if the heartbeat Interest or the Data packet is dropped on their way.
As presented earlier, the broker only sends the name of the matched data.
Therefore, the subscriber will be responsible for retrieving that matched data,
which follows the receiver-driven philosophy of the ICN paradigm.
Better Infrastructure Utilization
Two important features of the IDD layer are multicast and in-network caching
capability. Routers store all the incoming interests in a Pending Interest Table
Chapter 3. Data Dissemination in CVST 52
and, upon receiving of the data, satisfy all the Interest packets for the same
name at once. They also use their storage to save the content in their content
store to satisfy future Interest packets. As presented earlier, the broker will
send only the name of the data to the subscribers, and the subscribers will
request that data separately. The broker can satisfy the Interest packets of all
the subscribers with the same data packet, i.e. the broker sends out the data
only once and all the subscribers will receive it.
Scalability
Multiple broker instances may receive the published data for the system. This
scalability is achieved by using the same name for all of them. The best route
selection strategy will send the Interest packets towards the lowest-cost next
hop. Since the sequence number is set by the publisher and increasing over
time, one of the instances of the broker will receive the data.
To handle scalability at the broker, we use the fact that, in our design,
the subscriber will receive the name of the data from the broker and will send
another Interest packet to retrieve the matched data. Thus, the power of
forwarding subscribers to the right place is in the hand of the broker. The
best route selection strategy can also help with the forwarding of the Interest
packets for the matched data to the proper server. For example, the broker can
forward the subscriber to the publisher itself and avoid storing the data itself,
using the fact that data names are unique, and the network is responsible for
retrieving the data.
Chapter 3. Data Dissemination in CVST 53
Security
We must address two problems in the security domain. First, the broker must
accept data only from known publishers. No one should be able to inject data
into the system. Second, no one should be able to access the published data
without authorization.
Every Interest and Data packet are signed to ensure the provenance and
integrity of the data. In the CVST platform, the first problem is solved by
having the broker act as the trustworthy key management system. The bro-
ker issues certificates for publishers and subscribers in the registration phase.
These certificates are used to sign the Interest and Data packets. Through
the broker, everyone has access to each other’s public key and can validate the
provenance of the data.
To solve the second problem, we must use a shared key encryption algo-
rithm. The broker will issue the shared key and provides it to the publishers
and subscribers. Since the broker can validate the identity of the publishers
and subscribers, only the authorized users will have access to the key. The
publishers encrypt their data, and the broker and subscribers decrypt the data
with the key.
3.2 Broker Architecture
CVST collects data from many types of producers, however, the consumers
are typically interested in a portion of the published data. Therefore, CVST
uses a content based publish/subscribe paradigm. For example, the central
Chapter 3. Data Dissemination in CVST 54
Communicatoin Layer (IDD, IP, ...)
Registration/SchemaManager
SubscriptionTable
PublicationTable
DistributedMatching Engine
Publishers Subscribers
Message Queuing
Broker
Figure 3.4: High-level architecture of content-based publish/subscribe overIDD in CVST
database of the CVST platform is a subscriber that receives all the new data
updates from all the data sources. In another example, a drone incident is a
content type that is generated and disseminated in the platform. A subscriber
may only be interested in the drone incidents in a particular area. Fig. 3.4
shows the high-level architecture of the publish/subscribe system in CVST
platform. The system consists of publishers, subscribers, and the broker that
communicate using a communication layer.
Micro-service Abstraction
Using micro-services is an approach in software system design in which the
system is structured into smaller individual service units. Each service runs as
Chapter 3. Data Dissemination in CVST 55
API Type Description
/register XPUB Register a schema in the broker for publication/unregister XPUB Remove the schema from the broker/publish XPUB Publish new data based on a registered schema/subscribe XSUB Register a query in the broker for subscription/unsubscribe XSUB Remove a registered query from the broker/schemas XSUB Request for the list of registered schemas/schemas/<:id> XSUB Request for a specific schema using its id
Table 3.1: The APIs exposed by XPUB and XSUB services
an independent process and communicates with other services through APIs.
To be able to abstract the implementation of different communication pro-
tocols, the broker is divided into three services, XPUB, XSUB, and Matcher.
Fig. 3.5 shows how these services communicate with each other using a message
queuing system.
XPUB is responsible for communicating with the publishers and XSUB
with the subscribers. XPUB and XSUB hide the complexity of different com-
munication layers from the Matcher and provide the ability to add more proto-
cols without affecting other parts of the system. They define a set of APIs that
can be used by publishers and subscribers to talk to the system. They also hide
the complexity of the Matcher from the publishers and the subscribers, which
provides the ability to improve and replace the Matcher without affecting cur-
rent publishers and subscribers. XPUB and XSUB support multiple protocols.
For each protocol, a separate instance of XPUB or XSUB is started.
For example, an instance of XPUB may listen on any address such as
"ndn:/broker/xpub", "tcp://0.0.0.0:4040" or "http://broker:8080/broker/xpub",
as long as XPUB or XSUB has the protocol implemented.
Chapter 3. Data Dissemination in CVST 56
XPUBWorkers
XSUBWorkers
Communication Layer (IDD, IP, ...)
MatcherWorkers
Message Queuing
Figure 3.5: Design of the Broker: Abstraction of the complexity of differentsystem components
Schema Registration
Each publisher must first register itself with the system. Registering a pub-
lisher means the publisher must provide a schema for its data, which will be
used later by the broker to verify the structure of the incoming data from
the publisher. Schema will also be used by subscribers to define the criteria
of their subscription. The registration information of the publisher is saved
in the Publication Table. The publisher may also provide some additional
configuration in the registration process.
Subscriptions
For subscribers, registration involves providing a query based on data schemas.
These queries are based on the schema that publishers have registered in the
system. Subscribers also provide callback paths that are used by the broker to
send notifications about the newly published data that match the registered
Chapter 3. Data Dissemination in CVST 57
Publisher XPUB Matcher XSUB Subscriber
Registraion
Publication
Subscription
Match Notification
Data Retrieval
register schemastore schema
ok
ok
register query
store query
ok
okpublish data
match datadata is matched
send notification
data request
data
Figure 3.6: Sequence Diagram of the Content-Based Publish/Subscribe System
queries.
Message Queuing
All the communication between the components of the broker is facilitated
by using a Message Queuing system. Using message queuing separates the
different components of the system and facilitates distributing them among
the various machines. Message Queuing, itself, is a cluster of nodes which act
as one logical system from points of view of other components.
Matching Engine
The incoming data from a publisher is sent to the XPUB. The XPUB puts
the data in a queue which is then picked up by one of the Matcher’s workers.
At first, the matching engine, based on the Publication Table, checks if the
Chapter 3. Data Dissemination in CVST 58
data conforms to the schema provided by the publisher, if not the data is
rejected. Then the data is matched against the Subscription table. If a match
is found, the data is put back in another queue with additional data from the
subscription. The matched data is picked up from the queue by one of the
XSUB instances, which sends notifications to the subscribers. The matched
data is stored for later retrieval by the subscribers. The sequence diagram of
the publication, matching and data retrieval is depicted in Fig. 3.6.
3.2.1 Discussion
In this section, we discuss the advantages of abstraction and breaking down
of the broker functionality across multiple micro-services.
Agility
By having separate services for different tasks, development can be focused on
individual components independently. Each service can be updated or even
replaced without affecting the other parts of the system as long as there is
an instance in the system that provides the compatible APIs. For example,
matching engine can be updated or even replaced independently of XPUB and
XSUB.
Efficiency
Another advantage of micro-service based design is more efficient use of un-
derlying infrastructure. As discuss in Section 2.2, CVST is running on top of
an IaaS layer which can provide resources on demand. Therefore, each service
Chapter 3. Data Dissemination in CVST 59
W W W W W W
XPUB XPUB XPUB XSUB XSUB XSUB
Matcher Matcher Matcher
MQ MQ
Load Balancer
Publisher Publisher
Subscriber Subscriber
Figure 3.7: Scalability of the Broker with Micro-service design
can independently ask only for the resources it requires, which increases the
efficiency.
Scalability
The Micro-service design provides solutions that can scale well to mitigate
high traffic demands. Although the broker is logically one node, the high
traffic load will be distributed among different system components. Fig. 3.7
shows how the system may run in a distributed way. Each part of the system
can run as a set of instances, which are glued together by the message queuing
system.
Chapter 3. Data Dissemination in CVST 60
For example, an XPUB may consist of multiple instances behind a load
balancer listening on a particular network address. One of these instances will
receive the data from a publisher and sends it to the message queuing system.
Message queuing is itself a cluster of nodes. XPUB workers can connect to any
of the Message queuing nodes and store the data in the queue. The Message
queuing will notify the Matcher workers about the new data. The data is
replicated on the other nodes to protect the system against failure.
One of the available workers of the matching system will pick up the new
data from the queue and match it against the subscription queries. The match-
ing engine is also a cluster of nodes, and all the nodes in that cluster can match
data and registered queries. The workers of the Matcher may connect to any
of the nodes in the matching engine cluster to do the matching, and if there
is a matched subscription, the worker will store the data back in the message
queuing system for the XSUB instances.
Similarly, XSUB load is distributed between its instances by the message
queuing system. One of the workers of XSUB picks up this data and notifies
the subscribers about the new match. XPUB and XSUB do not keep any
state about their clients and expose a set of RESTful APIs. Also, the clients
do not keep any state about the instance of the service, with which, they are
communication. Furthermore, on the network level, using IDD ensures that
there are no connections made between clients and the instances.
Chapter 3. Data Dissemination in CVST 61
3.3 Implementation
In this section, we review some implementation details of different components
of the content-based publish/subscribe overlay.
3.3.1 Broker Implementation
In this section, we review the implementation details of different broker com-
ponents. As depicted in Fig. 3.5, the broker has three main components,
Matcher, XPUB, and XSUB. They communicate with each other using a Mes-
sage Queuing system. XPUB and XSUB communicate to the outside worlds
using the Communication Layer.
Message Queuing
For Message Queuing, we use RabbitMQ [56]. RabbitMQ is an open source
messaging system that is robust and easy to use. It runs on all major operating
system and supports a lot of developing platforms. It also provides clustering
and high availability features.
Matching Engine
The Matching Engine is responsible for checking if the incoming data satis-
fies the set of constraints defined by the queries registered by the subscribers.
Running Queries is one the main features of any database engine. However,
matching data against a query, i.e. a reverse query, is not a standard feature.
To have a fast, distributed and reliable Matching engine, we have chosen Elas-
Chapter 3. Data Dissemination in CVST 62
ticsearch [57]. Elasticsearch is a distributed search engine that provides the
reverse query capability known as Percolator [58]. We also used Elasticsearch
to store Publication and Subscription tables. When a publisher registers its
schema or a subscriber registers its query, the corresponding data will be saved
in Elasticsearch for later retrieval.
Matcher, XPUB and XSUB
Matcher, XPUB and XSUB are implemented in Python language. As discussed
in Section 3.2.1, each component runs as a set of independent processes that do
the same job in parallel. For example, the workers of Matcher pick up the data
from RabbitMQ and match them against the subscriptions using Elasticsearch
(Fig. 3.4) and then put the result back in another RabbitMQ queue.
3.3.2 Communication Layer
The system supports two communication protocols: IDD and HTTP. IDD
communication is based on the design discussed in Section 3.1. The HTTP
APIs provide a similar URL syntax as the IDD layer. Publishers and Sub-
scribers have the option to choose either of these protocols to communicate
with the broker.
Data Serialization
We use Apache Avro [59] as the data serialization system. Apache Avro is a
sub-project of Apache Hadoop [60] and uses a compact binary data format
with a rich data structure and integrates with many developing platforms,
Figure 3.25: Forwarding Information Base table after XPUB, XSUB and pub-lisher are started
1 [ Forwarder ] onIncomingInteres t f a c e=261 i n t e r e s t=/xpub/ pub l i sh / s t a r t /hw_sensor/%FEQ%AE2 [ ContentStore ] f i nd /xpub/ pub l i sh / s t a r t /hw_sensor/%FEQ%AE L3 [ ContentStore ] no−match4 [ Forwarder ] onContentStoreMiss i n t e r e s t=/xpub/ pub l i sh / s t a r t /hw_sensor/%FEQ%AE5 [ Forwarder ] onOutgo ingInterest f a c e=259 i n t e r e s t=/xpub/ pub l i sh / s t a r t /hw_sensor/%FEQ%AE6 [ Forwarder ] onIncomingInteres t f a c e=259 i n t e r e s t=/hw_sensor/data/%FEQ%AE7 [ ContentStore ] f i nd /hw_sensor/data/%FEQ%AE R8 [ ContentStore ] no−match9 [ Forwarder ] onContentStoreMiss i n t e r e s t=/hw_sensor/data/%FEQ%AE
10 [ Forwarder ] onOutgo ingInterest f a c e=261 i n t e r e s t=/hw_sensor/data/%FEQ%AE11 [ Forwarder ] onIncomingData f a c e=261 data=/hw_sensor/data/%FEQ%AE/%FD%01/%00%0012 [ ContentStore ] i n s e r t /hw_sensor/data/%FEQ%AE/%FD%01/%00%0013 [ Forwarder ] onIncomingData matching=/hw_sensor/data/%FEQ%AE14 [ Forwarder ] onOutgoingData f a c e=259 data=/hw_sensor/data/%FEQ%AE/%FD%01/%00%0015 [ Forwarder ] onIncomingInteres t f a c e=259 i n t e r e s t=/hw_sensor/data/%FEQ%AF16 [ ContentStore ] f i nd /hw_sensor/data/%FEQ%AF R17 [ ContentStore ] no−match18 [ Forwarder ] onContentStoreMiss i n t e r e s t=/hw_sensor/data/%FEQ%AF19 [ Forwarder ] onOutgo ingInterest f a c e=261 i n t e r e s t=/hw_sensor/data/%FEQ%AF20 [ Forwarder ] onIncomingData f a c e=261 data=/hw_sensor/data/%FEQ%AF/%FD%01/%00%0021 [ ContentStore ] i n s e r t /hw_sensor/data/%FEQ%AF/%FD%01/%00%0022 [ Forwarder ] onIncomingData matching=/hw_sensor/data/%FEQ%AF23 [ Forwarder ] onOutgoingData f a c e=259 data=/hw_sensor/data/%FEQ%AF/%FD%01/%00%00
Figure 3.26: Interests and Data packets log during XPUB and publisher com-munication
3.5 Evaluation and Performance Tests
In this section, we present some results of our system evaluation and perfor-
mance tests. In Section 3.5.1 we go over the network trace of publishing traffic
flow sensor data. In Section 3.5.2, we test the scalability of the workers of the
Matching Engine by putting the system under heavy load and then scale out
the workers by launching new virtual machines instances. In Section 3.5.3, we
test the performance of data delivery to subscribers using IP and Name-Data
Networking. All of these evaluations are end-to-end tests and involve all the
system components.
Chapter 3. Data Dissemination in CVST 79
Query Servers
Publications
Message Queuing
Worker VMs
....Figure 3.27: Scalability of the Matching Engine - Experiment Setup
3.5.1 IDD Publication Test
Fig. 3.25 lists the status of Forwarding Information Base table of the router
after XPUB, XSUB and Traffic Flow publisher are started. Line 1 is the
face registered by the XSUB, line 2 is the face registered by the Traffic Flow
publisher, and line 3 is the face registered by the XPUB service.
Fig. 3.26 lists the packet log of the start process discussed in Section 3.1.1.
Publisher periodically sends the “start” Interest packet to the XPUB instance.
Line 1 in Fig. 3.26 shows that the Interest packet of the “start” process is re-
ceived by the XPUB, under the name /xpub/publish/start/hw_sensor/%FEQ%AE.
/xpub is the path to the XPUB instance, /publish/start is the action verb
for the “start” process, /hw_sensor is the path of the publisher, and %FEQ%AE
is the binary format of the sequence number of the data available in the pub-
lisher.
Line 6 shows the Interest packet that the XPUB sends back to the publisher
at /hw_sensor/data/%FEQ%AE to request the newly published data. Notice
that the same sequence number is used in the name of the data. Line 11
Chapter 3. Data Dissemination in CVST 80
0
1000
2000
3000
4000
5000
6000
7000
8000
0 1000 2000 3000 4000 5000 6000 7000 0
2
4
6
8
10
12
14
16D
eliv
ery
Rat
e (m
sg/s
)
Num
ber o
f Wor
kers
Time (s)
workers
deliveries.mean(1m)
Figure 3.28: Scalability of the Matching Engine, one minute rolling average
show the data is sent by the publisher to XPUB, properly segmented. Line 12
indicates that data is cached so it will be available for another Interest packets
requesting the same data. Line 15 shows that after the XPUB receives the new
data, it immediately sends an Interest packet requesting for the next sequence
number.
3.5.2 Scalability of the Matching Engine
Since the Matching Engine does most of the computationally intensive job
in the system, we did an experiment to test how it can scale out. Fig. 3.27
shows the setup of the experiment. We used separate machines for different
components of this experiment, the Message Queuing Server, the Query Server,
and Worker Servers are each a separate virtual machine (VM) instance running
Chapter 3. Data Dissemination in CVST 81
0
1000
2000
3000
4000
5000
6000
7000
0 1000 2000 3000 4000 5000 6000 7000 0
2
4
6
8
10
12
14
16D
eliv
ery
Rat
e (m
sg/s
)
Num
ber o
f Wor
kers
Time (s)
workers
deliveries.mean(5m)
Figure 3.29: Scalability of the Matching Engine, five minutes rolling average
on the SAVI platform. Therefore, we can launch as many workers as we
need independently of other parts of the system. To keep workers busy, we
bombarded the message queuing system with new messages and kept it full
throughout the experiment. Then we started workers one by one to consume
the messages and match them against a subscription query in the Query Server.
Over time, the number of workers is increased from 1 to 16, and the rate of
the message delivery is measured every ten seconds on the Message Queuing
Server.
Fig. 3.28 and Fig. 3.29 show the message delivery rate of the message
queuing system over one minute and five minutes rolling average respectively.
To better understand the trend of the data we have superimposed the number
of workers over the delivery rate. The left axes in Fig. 3.28 and Fig. 3.29
Chapter 3. Data Dissemination in CVST 82
Subscribers
....R1 R2
Publications BrokerL1
Figure 3.30: Data usage: IDD vs IP — Experiment Setup
show the delivery rate, while the right axes indicate the number of workers
over time. The increasing trend of the delivery rate follows the number of
workers in the system. For example, when there is one worker, the delivery
rate is about 500 msg/s. Increasing the number of workers to five, increases
the delivery rate to 2500msg/s. This experiment shows the system can scale
out and mitigate a higher load with a higher delivery rate by adding more
parallel workers to the system.
3.5.3 IDD and IP Performance Comparison
Next, we setup a test experiment to evaluate the performance of IDD layer.
Fig. 3.30 shows the test setup. Similar to Section 3.5.2, we continuously publish
our test publications to the broker, and a setup a series of subscribers with
queries that match those publications. We have set up the system in a way that
all the communications between the broker and the subscribers are transmitted
over a single network link, noted as L1. As shown in Fig. 3.30, L1 is between
routers R1 and R2. R1 is the connection point of the broker’s network and
L1, and R2 is the connection point of the subscribers’ network and L1.
We tested the system with both IP and IDD as the communication pro-
tocol. Throughout the trial, we increase the number of subscribers every 60
seconds and measure the link utilization of L1 every 10 seconds. Fig. 3.31
shows the one minute rolling average of the link utilization of L1 when sub-
scribers are added to the system every 60 seconds. As one can see, when the
subscribers use IP, link utilization of L1 is constantly higher than when IDD
is used.
This difference comes from the fact that IDD only puts one copy of the
data on the wire. All the subscribers are sending Interest packets for the same
data name. After one of the subscribers sends its Interest packet over L1 to the
broker, the subsequent Interest packets are stored in R2. The broker satisfies
the first Interest packet with the matched data. This data reaches R2 on its
Chapter 3. Data Dissemination in CVST 84
path. R2 caches the data and satisfies all of its pending Interest packets. On
the other hand, in the IP-based communication, there is no in-network caching
on the protocol layer and the broker has to send each subscriber a new copy
of the data, which results in a higher link utilization.
To improve the performance of IP-based protocols one must put a cache
near the R2. Then, all the requests from subscribers towards the broker should
be rerouted to that cache, for example, by configuration in the application of
the subscribers or packet level inspection at R2. In IP, data is coupled with its
location, the application and routers. Therefore, the network lacks the sup-
port for mobility and provenance. In IDD, they are decoupled. Therefore, the
application is responsible for creating the content and the network is respon-
sible for delivery it. The application does not have to know about the client
or network configuration, and the network does not need to know about the
application specific packets to forward them to specific caches.
3.6 Summary
In this chapter, we presented our Naming design for Named-Data Networking
to have push notification in the data dissemination layer of CVST platform.
We discussed how the design would provide a simple, scalable communication
layer that has support for security and mobility. We have used this design in a
scalable and distributed implementation of a content-based publish/subscribe
system. Using micro-services provides the publish/subscribe system the capa-
bility to easily scale out and serve more requests. We also demonstrated some
Chapter 3. Data Dissemination in CVST 85
examples, such as live stream of drone events, using this dissemination layer
in the CVST platform.
Chapter 4
Content Delivery in Service
Providers
The CDN architecture is optimized to deliver the content until it reaches the
network of service providers. The Service Providers (SP) usually place Con-
tent Delivery Network (CDN) caches at the Internet Exchange peering points,
connected to the core of the network. Inside the operator’s network, it is a
different matter. The area of the network that is close to the consumer, and
is known as last mile network, is not optimized for Over-The-Top content.
The CDN architecture does not solve the inefficient use of the SP’s network
infrastructure. When the users are requesting the same content, that content
is transmitted over the network of the SP multiple times. In this chapter,
we use time-to-exhaustion (TTE) as our metric and formulate the problem to
place the caches in the network and route the content in a way that TTE is
maximized.
86
Chapter 4. Content Delivery in Service Providers 87
INTERNET CDNCDNCDNCDN
NetflixGoogleAmazonAkamai
3rd PartyCaches
OperatorCDN
Access{ { Last Mile
Agg/Edge
Figure 4.1: Network of a Service Provider
The rest of this chapter is organized as follows. In Section 4.1, the content
delivery problem in a service provider is investigated. Further, details of our
analytical model are discussed in Section 4.2. Simulation results and validation
are provided in Section 4.3.
4.1 Problem Definition
4.1.1 Content Distribution in Service Providers
Fig. 4.1 shows a simplified path that content takes from its source, through the
operator’s network and at last to the consumer. A service provider’s network
usually consists of multiple layers, the Core layer, the Aggregation layer, and
the Access layer. The core of the network transfers the highest volume of data
from various aggregation sites between sources and destinations. The Core
has few points of presence (PoP) and high capacity communication. Content
servers are usually connected to the core through an Internet exchange peering
point. The next level is the Aggregation level and is a concentration point of
multiple distribution centers, which themselves may be connected to smaller
distribution Edge centers. Each center in the aggregation level usually serves
Chapter 4. Content Delivery in Service Providers 88
INTERNET NetflixGoogleAmazonAkamai
PeeringPoint
Agg/Edge
(a) Content delivery from Peering Points
INTERNET NetflixGoogleAmazonAkamai
PeeringPoint
Cache
Agg/Edge
(b) Effect of caching on network traffic
Figure 4.2: Content distribution in Service Providers
about one to three million customers. The final layer is called Access layer and
is directly connected to the consumers. For example, a cable provider edge
layer contains cable modem termination systems (CMTS) and each CMTS
serves about 10 to 50 thousand subscribers. Access layer of a wireless service
provider contains several cellular antennas.
Now consider the subscribers that request an OTT video content. As shown
in Fig. 4.2a, for every request for content, a new connection is created between
the content source and the consumer’s machine. Even when all the users are
requesting for the same content, that content is transmitted over the network
multiple times. Note that the source of this content may be either controlled
by the operator itself or come from a VoD content server owned by a 3rd
party CDN. If the content source is a live stream from outside of the network,
operators are faced with an even bigger challenge than for VoD content. Many
consumers watch live stream content concurrently, and the operator does not
Chapter 4. Content Delivery in Service Providers 89
have any control over the content coming from outside of the network. This
structure is not scalable and is an apparent waste of underlying resources.
For example, consider a Service Provider in Canada that is serving OTT
content to its users in Ontario and Quebec. If the peering point is in Chicago,
all the traffic requests from users in Quebec and Ontario are served from
Chicago. Installing a cache in Toronto will save a lot of traffic that, other-
wise, would have gone over the network from Chicago to users in Ontario and
Quebec.
Fig. 4.2b shows a network that has a cache near the Aggregation and the
Edge level. All the flows, which were passing through the Core, are now
terminated at a lower level of the network. Therefore, putting a cache in lower
levels of the network saves the extra bandwidth used by multiple transmission
and increases the available capacity of the network. Hence, Content Providers
are putting their caches inside the operator’s network. Netflix, with its Open
Connect program, convinced the operators to set up cache servers even deeper
in their network, in places such as metro areas, to reduce the traffic load on
their core and peering points.
4.1.2 Time-to-exhaustion
In a network with increasing demand, such as service providers, congestion is
inevitable. For a service provider, serving content from a peering points incurs
cost. At the same time, serving more content to users means more revenue.
The demand increase will eventually exhaust the network at some point in the
future unless the onset of congestion of the network is increased. The network
Chapter 4. Content Delivery in Service Providers 90
S
S
S
D
D
Figure 4.3: Flows between sources and destinations pass through multiple links
onset of congestion is when the capacity of a link in the network is exceeded,
i.e., the link is congested.
However, the onset of congestion not only depends on the network topology
but also on the pattern of the growth of the demands. For example, network
congestion in a network with a linear demand growth is different from a net-
work with an exponential demand growth. Furthermore, the demand matrix
plays a major role in the onset of congestion of the network. For example,
introducing new services, offering new types of quality of service for content
delivery, or adding new customers change the onset of congestion of the net-
work.
Fig. 4.3 shows how content routing and caching can affect the onset of
congestion. Following the max-flow-min-cut theorem, the maximum amount
of flow passing from the sources to the destinations in a network is equal to
total link capacity of the minimum cut of that network. Now, consider a case
that most of the flows are routed through a critical link, which makes that
link congested sooner than later. Service providers have two solutions to this
problem. The first solution is optimizing the content routing and passing them
Chapter 4. Content Delivery in Service Providers 91
0 5 10 15 20 25 30Time (Months)
0
100
200
300
400
500
600
Traf
fic (G
B/s
)
Network Capacity
TTE 1 TTE 2
Demand 1Demand 2
Figure 4.4: Time-to-exhaustion. Traffic is increasing monthly until network iscongested.
through different links. Therefore, the critical link will have a lower average
load over time, and its congestion is delayed. Another option is to move the
flow destinations, e.g. caches, to other parts of the network. In other words,
putting caches in the network will delay the onset of congestion.
For example, assume that the demand is increasing every month. Fig. 4.4
shows such a scenario. If the current demand (shown as Demand 1) in the
network is 100Gb/s and the network can handle a maximum of 400Gb/s, the
current infrastructure will keep up with the traffic for the next 16 months.
However, by placing caches in the network and optimizing content routing,
the demand pattern changes (shown as Demand 2), and the network will stay
congestion-free for another eight months.
Chapter 4. Content Delivery in Service Providers 92
The problem that SPs are facing is how to plan their future network to
accommodate the constant increasing of the demand, to provide a congestion
free network and to minimize the costs. Another challenge is that the SP
already has an established network. SPs need time to buy pieces of equipment,
test them, and deploy them in their infrastructure. These investments keep
the network congestion-free for a limited time.
Also, the budget planning process has a time element. In other words,
the budget is planned for a limited period, such as a year. Therefore, Service
providers use the notion of time-to-exhaustion for forecasting. TTE becomes
crucial for network capacity planning since it affects the amount and the timing
of investment in the infrastructure. For example, with a limited budget, the
SP must choose how to plan the additional capacity and where to put the
caches and what type of content should to cached.
We aim to maximize the time-to-exhaustion, considering a limited budget,
by placing caches in the best locations and optimizing the content routing. We
will show that using ICN-based paradigms, such as Named-Data Networking,
will outperform optimal cache placement and content routing in CDN and will
prolong time-to-exhaustion of the network. The strategy layer of the NDN can
be used to route the content in the optimal way.
4.2 Problem Formulation
We model our network as a directed graph G(V,E) with the set of nodes V
and links E. U is the notation for the set of nodes that have a demand for
Chapter 4. Content Delivery in Service Providers 93
Constants
V Set of nodesE Set of directional linksG(V,E) Graph of the networkP Nodes that are connected to the peering pointsU Nodes that have a demand for contentsC Nodes that can cache contentsLk Size of the content kαki Demand for content k at node i
Γ+i ,Γ
−i Set of ingress and egress neighbors of node i
rki Maximum rate node i can read content k from its cacheB Total storage available for all cachesV (.) The function that maps storage to its budget valueci,j Capacity of link (i, j)I(.) Indicator function, 1 if the condition is true, 0 o.w.M Maximum number of caches in the networkφsdi,j Shortest-path betweenness of link (i, j) from node s to d
Common Variables
Si Storage at node ipi Decision variable for cache placement at node ihki Decision variable for caching content k at node iβki Total demand by node i for content k
CDN Specific Variables
fkdi,j Flow for content k on link (i, j) going to node dγkds Traffic flow from node s to node d for content k
NDN Specific Variables
fki,j The rate interests for content k is sent on link (i, j)
Table 4.1: Notations
Chapter 4. Content Delivery in Service Providers 94
contents. P indicates the set of nodes that can satisfy demands for contents,
e.g Internet exchange points. The set of nodes that are candidates for caching
contents is noted by C. All the notations are listed in Table 4.1.
4.2.1 Demands and Storage Budget
To find the TTE of the network, we will model the network for one time epoch.
We assume that within this time epoch, the demands are known and fixed,
but the location of the caches, the cached content and content routing are not.
We also assume a limited storage budget, B, is available for capacity planning
of all the caches in the network.
Since the demand of each user changes over time following different pat-
terns, we will run an exhaustive search to find the TTE by solving a series
of feasibility problems. A feasibility problem does not have any objective and
will only find a feasible solution to the problem. For each budget value, we
change the demands of the users following a pre-known pattern. If, for a set of
demands, the network becomes congested, the problem will become infeasible.
When the problem becomes infeasible, we have found the TTE of the network.
Final solution of the model provides a cache placement and content routing
policy that maximizes the TTE of the network.
Demands
We denote the demand at each node i for content k by αki . Note that αk
i
depends on time, however, we are solving the problem for each time epoch
separatly. The demand at each node also depends on whether the node i
Chapter 4. Content Delivery in Service Providers 95
caches content k or not, denoted by a binary variable hki . In other words, the
traffic of populating a cache is also a demand. Therefore, total demand at
node i can be written as:
βki = αk
i + hki ∀i ∈ C ∪ U (4.1)
Note that βki is the number of the requests for content k, not the size of
the demand. The size of the demand is Lkβki where Lk is the size of content
k.
Storage Budget
Each cache in the network is assigned a part of the storage, denoted by Si.
However, B is the storage budget in dollar value. We assume that the relation
between the amount of storage and its dollar value can be written as a function
V (Si). The sum of all the budgets assigned to caches should be equal to B.
Let pi be the binary variable that decides if node i is a cache. Therefore, the
budget constraint can be written as Eq (4.2).
∑i
piV (Si) ≤ B ∀i ∈ C (4.2)
Here we assume that V (.) is a linear function, however, this can be extended
to any convex function. For each cache, total cached objects can not exceed
the size of the storage of that cache as written in Eq (4.3).
∑k
Lkhki ≤ piSi ∀i ∈ C (4.3)
Chapter 4. Content Delivery in Service Providers 96
Also, total number of caches placed in the network can be limited by an
upper bound, M , as written in Eq (4.4).
∑i
pi ≤M ∀i ∈ C (4.4)
To have homogeneous caching, we may also add a constraint that enforces
all Si to be equal.
Cache Replacement Policy and Routing
Caching policy provided by the solution will maximize the TTE of the network.
hki is the binary variable that shows if content k is cached at node i. Solving
the model for two different time epochs with different demands will result
in different hki . The difference between hki for different demands will be the
cache replacement policy of node i. Adopting a certain caching replacement
policy, such as Least Recently Used (LRU) or Least Frequently Used (LFU),
will reduce the TTE of the network.
The solution also provides the content routing policy for the network.
Adopting a routing protocol such as shortest-path will also reduce the TTE of
the network. We study this effect in the result section.
4.2.2 Content Delivery Networks
In service providers, transparent caching is done by putting one or more caches
in the network and re-routing the requests towards them. SPs may also host
the content sources of their own or from third parties. To model this, we define
Chapter 4. Content Delivery in Service Providers 97
a multi-commodity flow problem.
Flow Conservation
The flow conservation at node s for content k can be written as Eq (4.5). We
denote fkdi,j as the flow for content k on link (i, j) going to node d and γkds for
the flow for content k from node s to node d. The left-hand side of Eq (4.5)
is the difference between total egress (Γ−s ) and ingress (Γ+s ) flows for content
k at node s that is destined for node d.
∑j∈Γ−
s
fkds,j −
∑j∈Γ+
s
fkdj,s = γkds − Lkβ
ks δ(s− d) ∀s, d ∈ V (4.5)
The right-hand side of Eq (4.5) is the total flow that is originated at node s
towards node d for content k minus the demand at node s for content k. δ(i)
is the Kronecker delta function, it is equal to 1 when i is zero, otherwise it is
zero. Therefore, Lkβks in Eq (4.5) will only have any effect when s and d are
the same node. In other words, the ingress and egress flow destined to node
d at any node other than d is equal to the traffic produced at that node for
node d. When s and d are equal all ingress traffic into node d will be equal to
the demand at node d. Therefore, considering the fact that node d does not
send traffic to itself (i.e. fkdd,j = 0,∀j and γkdd = 0), Eq (4.5) will be reduced to
∑j∈Γ+
d
fkdj,d = Lkβ
kd
Chapter 4. Content Delivery in Service Providers 98
Cache Population Traffic
The cache population traffic is satisfied by peering points. Therefore, the total
demand originated at the core (P) of the network, must be bigger than the
size of the cached content (Eq (4.6)).
∑s∈P
γkis ≥ Lkhki ∀i ∈ C (4.6)
I/O and Link Capacity Limits
A node can only become a source of the flow for a content request when it
is a cache and it has the content cached. I(i ∈ C) in Eq (4.7) is equal to 1,
if only node i is a cache candidate. hki will be equal to 1 when the content
k is cached at node i. rki is the rate that each node can read contents from
its cache storage and put them on the wire. It is the limitation of the node’s
hardware, e.g. I/O limit of the node’s hard disks.
∑d
γkdi ≤ I(i ∈ C)rki Lkhki (4.7)
Also, each link (i, j) has a limited capacity, denote by ci,j. The link capacity
enforces that the sum of all the flows to all destinations for all contents be less
than total link capacity, as in Eq (4.8).
∑k,d
fkdi,j ≤ ci,j (4.8)
The complete feasibility problem that models a CDN in the network of a
service provider is shown in Fig. 4.5:
Chapter 4. Content Delivery in Service Providers 99
solvesubject to∑
j∈Γ−s
fkds,j −
∑j∈Γ+
s
fkdj,s = γkds − Lkβ
ks δ(s− d) ∀s, d
∑s∈P
γkis ≥ Lkhki ∀i ∈ C∑
d
γkdi ≤ I(i ∈ C)rki Lkhki ∀i ∈ V \ P∑
k,d
fkdi,j ≤ ci,j
βki = αk
i + hki ∀i ∈ V \ P∑i
piV (Si) ≤ B∑k
Lkhki ≤ piSi
Figure 4.5: Feasibility model for CDN
Shortest-path routing
Routing in the network of the service providers is usually based on shortest-
path routing. To study the effects of shortest-path routing, we add a routing
constraint to our model. Shortest-path routing is modeled using the shortest-
path betweenness centrality of each link.
Betweenness centrality (BC) is one of the centrality metrics in graphs [62].
Betweenness centrality measures the degree to which a node or a link is needed
when connecting other nodes along paths. Shortest-path betweenness central-
ity of the link (i, j) with respect to the source node s and the destination
node d, denoted as φsdi,j, is defined as the proportion of the number of the
Chapter 4. Content Delivery in Service Providers 100
shortest paths from node s to d that passes through link (i, j). Therefore, the
average traffic for content k that passes through link (i, j) from source s to
destination d can be written as φsdi,jγ
kds . To model shortest path routing we can
add Eq (4.9) to the model. Eq (4.9) will have the link (i, j) to not transfer
any traffic more than its share, if the routing is done using shortest-path.
fkdi,j ≤
∑s
φsdi,jγ
kds s ∈ V (4.9)
4.2.3 Named-Data Networking
Interest Forwarding
To model NDN, we will find the locations that potentially can satisfy more
interest in contents. This notion of interest here is more of the nature of
content popularity in a node, similar to the virtual interest packets studied
in [63], and is different from the Interest packet in NDN paradigm. We denote
fki,j as the rate that interest for content k is forwarded on link (i, j). Since NDN
is a point-to-point protocol we do not have flows from sources to destinations,
but potential interests that move around the network until they are satisfied.
Suppose node s has some interest in content k (βks ). Therefore, the egress
interests (∑
j∈Γ−sfks,j) from node s is increased by βk
s . This is written as an
inequality in Eq (4.10).
∑j∈Γ−
s
fks,j −
∑j∈Γ+
s
fkj,s ≤ βk
s (4.10)
Now consider a node that has a content cached in its content store and
Chapter 4. Content Delivery in Service Providers 101
can satisfy interest for that content and remove the interest from the network.
Each node also has an I/O limit for reading its content store that limits the
rate interests are satisfied. Otherwise, the interest will be forwarded towards
other nodes in the network. Therefore, a node can at most satisfy the interests
by the rate that is bounded by its I/O limit, as written in Eq (4.11).
∑j∈Γ−
s
fks,j −
∑j∈Γ+
s
fkj,s + I(i ∈ C)rksh
ks ≥ βk
s (4.11)
Consider the scenario that node s is not caching content k. Therefore,
Eq (4.10) and Eq (4.11) will be reduced to an equality and will enforce that
node s forwards all of its ingress and local interests. However, if node s caches
content k, the ingress interests can be satisfied by an amount bounded by
the hardware limitations of node s. Finding the movement of this potential
interest in the network can be used to find the best place to cache the content.
Link capacity limit
The next step is to model the link capacity constraint. In NDN, Data packets
follow the reverse path of the Interest packet to reach the destination. There-
fore, sending an interest over the link (i, j) will result in the data sent back over
the link (j, i). We can use this to write link capacity constraint as Eq (4.12).
∑k
Lkfki,j ≤ cj,i (4.12)
Including Eq (4.1), Eq (4.2), Eq (4.3), the complete feasibility problem for
NDN is shown in Fig. 4.6
Chapter 4. Content Delivery in Service Providers 102
solvesubject to∑
j∈Γ−s
fks,j −
∑j∈Γ+
s
fkj,s ≤ βk
s∑j∈Γ−
s
fks,j −
∑j∈Γ+
s
fkj,s + I(i ∈ C)rksh
ks ≥ βk
s∑k
Lkfki,j ≤ cj,i
βki = αk
i + hki ∀i ∈ V \ P∑i
piV (Si) ≤ B∑k
Lkhki ≤ piSi
Figure 4.6: Feasibility model for NDN
4.3 Results
We evaluated the numerical result of our model using multiple network topolo-
gies. Fig. 4.7 is one of the Rocketfuel networks [64]. Fig. 4.8 is a Dorogovtsev-
Goltsev-Mendes (DGM) topology and Fig. 4.9 is a tree network. The Rock-
etfuel topology is simplified by removing the leaf nodes from the original net-
work and consolidating the demands from the removed nodes into their parent
nodes [65]. The simplified network has 50 nodes and 194 directed links. These
three topologies are comparable in size. The number of users is 25 nodes in
Rocketfuel and 27 nodes in DGM and tree topologies. We consider one peering
point for each network, and the rest of nodes are cache candidates nodes. At
each node, the demand for each content follows a Zipf distribution with α = 2.
We assumed all the users have the same demand, and it is uniformly increasing
Chapter 4. Content Delivery in Service Providers 103
1
2
3
4
5
6 7
8
9
10 11
12
13
14
15
16
1718
19
20
2122
23
24
25
26
27
28
293031
3233
34
3536
37
38
39
40
41
42
4344
45
46 47
48
49
50
Figure 4.7: Rocketfuel network
by 5% every month. This increase is based on the current observation of OTT
demand increase. As mentioned in Section 4.2.1, for each budget point we
solve a series of feasibility problem and increase the demand until the network
is saturated. We also simulated the back-pressure algorithm in [63] to compare
with the performance of our model.
4.3.1 Time-to-Exhaustion of different topologies
To evaluate the performance of the CDN method, we find the TTE of the
network by putting at most four caches. The assigned storage budget is equally
divided between these nodes, assuming they are all using similar hardware. In
other words, we use homogeneous caching. For example, in Rocketfuel network
(Fig. 4.7), Nodes 5, 10, 12 and 14, are selected for caching, and respectively,
in DGM topology, Nodes 2, 3, 4, 5 and tree topology, Nodes 2, 3 and 4 are
Chapter 4. Content Delivery in Service Providers 104
1
23
45
6
78
910
11
12
13
14
15
1617
18192021
2223
24
25
26
27
28
29
30
31
32
33
34
35
36
37
3839
40
4142
Figure 4.8: DGM network
selected for caching. It is also worth noting that in the tree topology only three
nodes are selected for caching, since adding more caches has not increased the
TTE further. To evaluate the performance of NDN, we will enable caching
in all the candidate nodes. Therefore, storage budget will be equally divided
between more nodes, and each node can cache less number of objects.
Fig. 4.10, Fig. 4.11 and Fig. 4.12 show the TTE in different topologies
while using CDN model, NDN model and NDN simulation using back pressure
algorithm. We assumed that there is demand for 2000 objects, divided into 100
popularity groups, each with the size of 1Mb and all the links in the network
have the capacity of 1Gb/s. We had placed at most four caches in CDN while
all the nodes can cache in NDN scenarios. Note that in all the topologies,
NDN-Simulation using back-pressure closely follows our NDN-model.
At very low storage budget, CDN and NDN had a similar TTE, because
most of the content is provided by the peering point, and that will become
the bottleneck of the network. This means network onset of congestion will be
Chapter 4. Content Delivery in Service Providers 105
1
2
3
4
5
6
7
8
9
10
11
1213
1415
16
17
18
19
20
2122
23
24
25
26
27
28
29 30
31
32
33
34
35
36
37383940
Figure 4.9: Tree network
similar for both NDN and CDN scenarios. Different topologies have different
TTE for very low storage budget. The TTE depends on the onset of con-
gestion, and the onset of congestion depends on the topology of the network.
TTE is lowest for the tree topology and the highest for the DGM topology.
This observation is also in agreement with the reciprocal of network criticality
of each topology [66].
By increasing the caching storage, TTE is also increased. The storage
budget is equally divided between all caches. Therefore, the increase in the
total storage budget will increase the TTE. As the number of caches increases,
each cache will receive a smaller portion of the budget. Therefore, when there
is not enough additional storage available to each cache, there will be no
change in the number of cached contents, and the TTE will not change either.
This minimum increase in storage depends on the number of caches in the
network. In NDN, the steps are larger since there are more caches and a
greater increase in the total storage budget is required to cache more contents.
Chapter 4. Content Delivery in Service Providers 106
0
10
20
30
40
50
60
70
80
90
2 4 6 8 10 12 14 16 18 20
Tim
e to
Exh
aust
ion
(Mon
ths)
Cache Storage Budget (Gbit)
NDN-ModelNDN-Simulation
CDN
Figure 4.10: Time-to-exhaustion in Rocketfuel network
In CDN, the steps are smaller since there are only four caches and a smaller
amount of increase in storage budget, compared to NDN, will result in more
cached contents. However, the height of the steps decreases with increase of
the budget, because caching begins losing its effect. There is also a limit on
the maximum TTE of each topology, after which even caching does not help
anymore. This TTE is the maximum that a network can reach with the help
of caching. Similar to the low budget TTE, the maximum TTE also depends
on the topology of network.
Furthermore, in low storage budget, there is little difference in TTE be-
tween using CDN and NDN. Because of homogeneous caching, sometimes CDN
even performs better. However, in all the topologies, the network that uses
Chapter 4. Content Delivery in Service Providers 107
0
10
20
30
40
50
60
70
80
90
2 4 6 8 10 12 14 16 18 20
Tim
e to
Exh
aust
ion
(Mon
ths)
Cache Storage Budget (Gbit)
NDN-ModelNDN-Simulation
CDN
Figure 4.11: Time-to-exhaustion in DGM network
CDN is saturated in much lower storage budget compared to NDN. This bet-
ter performance is the direct result of the NDN paradigm. In NDN, due to
its in-network caching and point-to-point nature, each cached content is sent
over the links only once. However, in CDN each content is sent multiple times.
This waste of link capacity shows itself by having the network saturated much
sooner. There is a huge difference in maximum TTE between using CDN or
NDN in each topology. In Rocketfuel, using CDN will saturate the network
after 47 months. But using NDN, the network can be operational until 77
months. Similarly, DGM with CDN is operational for 74 months and with
NDN for 82 months. Tree topology with CDN is operational for 27 months
and with NDN for 67 months. This huge difference in tree topology is because
Chapter 4. Content Delivery in Service Providers 108
0
10
20
30
40
50
60
70
80
90
2 4 6 8 10 12 14 16 18 20
Tim
e to
Exh
aust
ion
(Mon
ths)
Cache Storage Budget (Gbit)
NDN-ModelNDN-Simulation
CDN
Figure 4.12: Time-to-exhaustion in Tree network
in NDN caches are placed throughout the network. As in Fig. 4.9, NDN places
caches in Nodes 2 to 13. But CDN only places caches in Nodes 2, 3 and 4. For
example, having a cache in Node 5 will reserve bandwidth in all the up-links
and will make more capacity available to deliver more content.
4.3.2 Limited NDN Deployment
To see how much of the difference in TTE between CDN and NDN comes from
the number of caches in the network, we will limit the number of caches in
NDN to four. Fig. 4.13 shows that even with four caches, content delivery using
NDN outperforms the CDN design. We have also considered a non-practical
case that every node in the CDN can also cache contents. This case is just for
Chapter 4. Content Delivery in Service Providers 109
0
10
20
30
40
50
60
70
80
0 5 10 15 20
Tim
e to
Exh
aust
ion
(Mon
ths)
Cache Storage Budget (Gbit)
NDN-full-cacheNDN-limited-cache
CDN-full-cacheCDN-limited-cache
Figure 4.13: Changes in TTE of Rocketfuel topology with number of caches
the comparison and in practice cannot be implemented due to the nature of
CDN. One could say that one of the reasons behind the NDN proposal is the
impossibility of in-network caching in TCP/IP. However, even if all the nodes
in the CDN had the caching capability, the network will saturate similar to
the case that there are four caches in the network. In addition, limited NDN
deployment has better TTE for low budget than full NDN deployment. This
suggests limiting the number of NDN caches when the storage budget is low.
We can also look at link utilization in the network. Fig. 4.14 shows the
percentage of links with various percentage of utilization during network con-
gestion. Using CDN, more than 60% of the links will have a link utilization
of more than 90%. In contrast, NDN scenario, even with limited deployment,
Chapter 4. Content Delivery in Service Providers 110
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
CDN-limited
CDN-Full
NDN-limited
NDN-full
<10 <20 <30 <40 <50 <60 <70 <80 <90 <100
Figure 4.14: Link utilization of NDN vs CDN
has less than 20% of the links with high utilization. Using NDN, has resulted
in a network that more than 40% of the links have link utilization of less than
10%. This difference in link utilization means that if CDN is used to increase
the TTE of the network, we have to increase the capacity of most of the links.
But using NDN will result in a much fewer bottlenecks, which makes capacity
planning much easier and cheaper.
4.3.3 I/O Speed Effect
One of the parameters we have considered in our modeling is the I/O limit of
each cache. The I/O limit depends on hardware design of the cache. Fig. 4.15
shows the effect of this parameter. To better see the difference the I/O speed
makes, we have increased the capacity of all the links to limit the effect of
congestion. As shown in Fig. 4.15, as the I/O limit increases from 10Gb/s
Chapter 4. Content Delivery in Service Providers 111
120
125
130
135
140
145
150
155
160
165
0 5 10 15 20 25 30 35 40
Tim
e to
Exh
aust
ion
(Mon
ths)
Cache Storage Budget (Gbit)
NDN-high-ioNDN-low-io
CDN-high-ioCDN-low-io
Figure 4.15: Changes in TTE of Rocketfuel topology with I/O limit
to 100Gb/s, TTE also increases. But it must be said that having low link
capacity will greatly diminish the improvement gained by having a hardware
with higher I/O limit.
4.3.4 Routing Protocol Effect in CDN
As mention above, our modeling tries to maximize the TTE and therefore
optimizes the routing of data. However, in practice routing is not optimal.
As shown in Fig. 4.16, by enforcing shortest-path routing for CDN in the
Rocketfuel network the TTE will be reduced by more than ten months. The
NDN does not have this problem since its strategy layer can employ an optimal
routing algorithm.
Chapter 4. Content Delivery in Service Providers 112
95
100
105
110
115
120
125
130
135
140
145
0 2 4 6 8 10 12
Tim
e to
Exh
aust
ion
(Mon
ths)
Cache Storage Budget (Gbit)
CDN-optimalCDN-SP
Figure 4.16: Changes in TTE of Rocketfuel topology with Routing algorithm
4.3.5 Heterogeneous Caching
Heterogeneous caching is using caches on the network that each has a different
amount of storage. In contrast to homogeneous caching, where all the caches
use the same amount of storage. Homogeneous caching may be cheaper since
the cache hardware comes in pre-configures packages, and having a customized
hardware costs more. Therefore, Service Providers must do a cost-benefit
analysis on having a heterogeneous caching system.
Fig. 4.17 shows the effect of heterogeneous caching on the TTE when NDN
is used. In using heterogeneous caching, the model will assign each cache
a different storage capacity while satisfying total storage budget constraint.
It is expected that the symmetry in tree and DGM topologies would imply
Chapter 4. Content Delivery in Service Providers 113
115
120
125
130
135
140
145
150
155
160
165
0 5 10 15 20
Tim
e to
Exh
aust
ion
(Mon
ths)
Cache Storage Budget (Gbit)
Rocketfuel-HTRocketfuel-HO
DGM-HTDGM-HOTree-HTTree-HO
Figure 4.17: Heterogeneous vs Homogeneous caching storage in NDN
little benefit to the heterogeneity. However, there is some difference in the
Rocketfuel topologies, which is less symmetric than other topologies we mod-
eled. If heterogeneous caching is employed, TTE in the Rocketfuel network is
increased at most by three months.
4.4 Summary
Service providers are under a lot of pressure due to daily increase of Over- The-
Top contents. In this chapter, we presented a cache placement and content
routing method for service providers to delay the congestion of their network
considering their limited budget. We modeled both ICN and CDN and aimed
to maximize the time-to-exhaustion of the network. Our result shows that
Chapter 4. Content Delivery in Service Providers 114
a limited deployment of ICN improves the time-to-exhaustion of the network
and lowers the number of links with high utilization.
Chapter 5
Conclusion
The Internet is evolving fast, regarding architecture and usage. Numerous
devices are getting connected to the Internet every day, and more and more
contents are constantly created. The current end-to-end communication us-
ing TCP/IP is not designed for these new use-cases. However, networking
paradigms, such as Information-Centric Networking, aim to tackle these prob-
lems. They move towards a point-to-point communication model, decouple
data names from their location and change router buffers into caches for con-
tent storage. In this chapter, we first review our contributions in this work and
then propose some ideas that can be implemented to extend our contributions.
5.1 Contributions
In this work, we designed a content-based publish/subscribe system using ICN
paradigm as the data dissemination layer in the CVST platform. Also, we
showed the benefits of using ICN in content delivery in service providers.
115
Chapter 5. Conclusion 116
5.1.1 Data Dissemination in CVST
The CVST platform collects a rich set of data from many transportation data
sources. These sources include traffic sensors, road cameras, road incidents
and closures reports, Twitter traffic reports, public transit data (bus location
information and bike station data), border delay time, and last but not least
the loop detector data.
We presented a content-based publish/subscribe system for CVST that
employs the ICN paradigm. In a content-based publish/subscribe systems, a
subscriber can define a query in addition to the topic of the interest and receive,
in real-time, the contents that match that query. We present the architecture
for a distributed broker that connects publishers and subscribers, registers the
schema for the publishers and saves the queries submitted by the subscribers.
These tasks are exposed as a set of APIs to publishers and subscribers. The
broker uses a set of scalable micro-services and supports ICN-base and IP-
based protocols to communicate with publishers and subscribers.
The publisher-broker and subscriber-broker communication layer over ICN
provides a platform to build an efficient, robust, scalable, and secure data dis-
semination layer. We presented the detailed design of the data dissemination
layer and its advantages. The platform has been used to publish live video
feed from drone flights, as well as many other data types. Our demonstration
shows the feasibility of Vision as a Service in an application platform.
Chapter 5. Conclusion 117
5.1.2 Time to Exhaustion
We proposed an in-network caching strategy for service providers to increase
the time-to-exhaustion of their network. We suggested that service providers
use Information-Centric Networking for caching and content delivery. Even a
limited deployment of ICN provides a substantial increase in time-to-exhaustion
of the network and lowers the number of links with high utilization. We studied
different parameters that affect the performance of content delivery, such as
I/O limited, routing algorithms, and heterogeneous and homogeneous caching.
We also validated our model by simulation.
5.2 Future Works
In this section, we review possible extensions of our work. The extensions are
in two categories, the extensions of the data dissemination layer for CVST and
the extensions of the content delivery in service providers using ICN paradigm.
We demonstrated the data dissemination layer uses the schema of a data
types for publications and subscriptions. More data sources can be easily
added to the data dissemination layer. Since the system understands the data
based on its schema, adding more data types is just creating and registering
the schema in the system. Access control, security, and privacy has native
support in Named-Data Networking. Our publish/subscribe system can easily
be extended to use these features to do verification, encryption, and authoriza-
tion. The broker can act as the central authority to control, issue and validate
the signing and encryption keys.
Chapter 5. Conclusion 118
We also expect that Vision as a Service (VaaS) will be available region-
wide by placing a network of drones throughout a region. Drones will be
dispatched on demand directly from base locations or transported by vehicle
to appropriate launching locations to investigate network anomalies.
The broker can also be extended to support aggregation queries. In the
current design, an application can subscribe to the raw data by filtering based
on some conditions. However, the data aggregation is a common feature in the
IoT systems. The Matching Engine micro-service is a good candidate to im-
plement the aggregation. The Matching Engine workers have direct access to
the data, and the queries and can use the Query servers as a temporary buffer
for both spatial and temporal aggregation. Additional micro-services may be
added to check for the aggregation result and notify the subscribers. The in-
terface for creating the aggregation queries can be implemented by extending
the Subscription portal.
Also, the Subscription Portal may be extended to register remote sub-
scribers. Currently, portal acts as a subscriber and receives notifications for
all of its registered subscriptions. However, the portal can register queries
for remote subscribers given their callback paths. The callback path of the
subscribers can be entered by the user in the query registration page of the
portal, and the portal will pass that information to the XSUB API.
Our work in controlling time-to-exhaustion in service providers may be
extended by integrating the logic of cache placement and content routing in a
central controller, such as SDI controller that SAVI provides. This controller
may use the output solution to the optimization problem and set the routing
Chapter 5. Conclusion 119
tables in the network routers to maximize the time-to-exhaustion. The solution
also provides a cache placement recommendation that may be used with the
dynamic resource allocation that an infrastructure such as SAVI provides to
instantiate new cache instances and route traffic towards that instances.
Also, different caching hardware may be used in various parts of the net-
work. Netflix OpenConnect [50] provides caching servers with different capa-
bilities, and the service providers must put the hardware in the best place in
the network. Our work can be extended to take into account these hardware
differences.
Bibliography
[1] D. Perino and M. Varvello, “A Reality Check for Content CentricNetworking,” in Proceedings of the ACM SIGCOMM Workshop onInformation-centric Networking, ser. ICN ’11. New York, NY, USA:ACM, 2011, pp. 44–49.
[2] V. Jacobson, D. K. Smetters, J. D. Thornton, M. F. Plass, N. H. Briggs,and R. L. Braynard, “Networking Named Content,” in Proceedings of the5th International Conference on Emerging Networking Experiments andTechnologies, ser. CoNEXT ’09. New York, NY, USA: ACM, 2009, pp.1–12.
[3] D. Raychaudhuri, K. Nagaraja, and A. Venkataramani, “Mobilityfirst:A Robust and Trustworthy Mobility-Centric Architecture for the FutureInternet,” SIGMOBILE Mob. Comput. Commun. Rev., vol. 16, no. 3, pp.2–13, Dec. 2012.
[4] A. Leon-Garcia, H. Bannazadeh, and A. Tizghadam, “Smart city plat-forms on multitier Software-Defined infrastructure cloud computing,” in2016 IEEE International Smart Cities Conference (ISC2) (ISC2 2016),Trento, Italy, Sep. 2016.
[5] J.-M. K.-M. Kang, H. Bannazadeh, and A. Leon-Garcia, “SAVI testbed:Control and management of converged virtual ICT resources,” in Inte-grated Network Management (IM 2013), 2013 IFIP/IEEE InternationalSymposium on, May 2013, pp. 664–667.
[6] “Global Internet Phenomena,” Sandvine, Tech. Rep.,2013. [Online]. Available: https://www.sandvine.com/trends/global-internet-phenomena/
[7] C. Labovitz, “Massive Ongoing Changes in Con-tent Distribution,” http://blog.streamingmedia.com/wp-
[8] C. Labovitz, S. Iekel-Johnson, D. McPherson, J. Oberheide, and F. Jaha-nian, “Internet inter-domain traffic,” ACM SIGCOMM Computer Com-munication Review, vol. 41, no. 4, pp. 75–86, 2011.
[9] World Urbanization Prospects: The 2014 Revision, Highlights (ST/E-SA/SER.A/352), United Nations, Department of Economic and SocialAffairs, Population Division, 2014.
[10] J. M. Hernandez-Munoz, J. B. Vercher, L. Munoz, J. A. Galache,M. Presser, L. A. H. Gomez, and J. Pettersson, “Smart cities at the fore-front of the future internet,” in The Future Internet Assembly. Springer,2011, pp. 447–462.
[11] A. Shariat, A. Tizghadam, and A. Leon-Garcia, “An ICN-Based Publish-Subscribe platform to deliver UAV service in smart cities,” in 2016 IEEEConference on Computer Communications Workshops (INFOCOM WK-SHPS): SmartCity16: The 2nd IEEE INFOCOM Workshop on SmartCities and Urban Computing (SmartCity’16), San Francisco, USA, Apr.2016.
[12] ——, “Optimizing time to exhaustion in service providers us-ing Information-Centric networking,” in 28th International TeletrafficCongress (ITC 28), Wurzburg, Germany, Sep. 2016.
[13] T. Koponen, M. Chawla, B.-G. Chun, A. Ermolinskiy, K. H. Kim,S. Shenker, and I. Stoica, “A Data-Oriented (and beyond) Network Archi-tecture,” SIGCOMM Comput. Commun. Rev., vol. 37, no. 4, pp. 181–192,Aug. 2007.
[14] D. Smetters and V. Jacobson, “Securing Network Content,” Tech. Rep.,2009.
[15] M. Gritter and D. R. Cheriton, “An Architecture for Content Rout-ing Support in the Internet,” in Proceedings of the 3rd Conference onUSENIX Symposium on Internet Technologies and Systems - Volume 3,ser. USITS’01. Berkeley, CA, USA: USENIX Association, 2001, pp. 4–4.
[17] N. Fotiou, P. Nikander, D. Trossen, and G. C. Polyzos, “Developing In-formation Networking Further: From PSIRP to PURSUIT,” BroadbandCommunications, Networks, and Systems, pp. 1–13, 2012.
[18] D. Lagutin, K. Visala, and S. Tarkoma, “Publish/Subscribe for Internet:PSIRP Perspective.” Future Internet Assembly, vol. 84, 2010.
[24] Named Data Networking. [Online]. Available: http://named-data.net
[25] G. Carofiglio, G. Morabito, L. Muscariello, I. Solis, and M. Varvello,“From Content Delivery Today to Information Centric Networking,” Com-put. Netw., vol. 57, no. 16, pp. 3116–3127, Nov. 2013.
[26] D. Rossi and G. Rossini, “On Sizing CCN Content Stores by Exploit-ing Topological Information,” in Computer Communications Workshops(INFOCOM WKSHPS), 2012 IEEE Conference on, March 2012, pp. 280–285.
[27] H. Yuan and P. Crowley, “Experimental Evaluation of Content Distribu-tion with Ndn and Http,” in INFOCOM, 2013 Proceedings IEEE, April2013, pp. 240–244.
[28] M. Varvello, D. Perino, and L. Linguaglossa, “On the Design and Im-plementation of a Wire-Speed pending Interest Table,” in Proceedings ofthe 2nd IEEE International Workshop on Emerging Design Choices inName-Oriented Networking, NOMEN, vol. 13, 2013.
[29] C. Dannewitz, M. D’Ambrosio, and V. Vercellone, “Hierarchical DHT-Based Name Resolution for Information-Centric Networks-Based NameResolution for Information-Centric Networks,” Comput. Commun.,vol. 36, no. 7, pp. 736–749, Apr. 2013.
[30] K. V. Katsaros, N. Fotiou, X. Vasilakos, C. N. Ververidis, C. Tsilopoulos,G. Xylomenos, and G. C. Polyzos, “On Inter-Domain Name Resolutionfor Information-Centric Networks,” in Proceedings of the 11th Interna-tional IFIP TC 6 Conference on Networking - Volume Part I, ser. IFIP’12.Berlin, Heidelberg: Springer-Verlag, 2012, pp. 13–26.
[31] A. Badam, K. Park, V. S. Pai, and L. L. Peterson, “Hashcache: CacheStorage for the Next Billion,” in Proceedings of the 6th USENIX Sympo-sium on Networked Systems Design and Implementation, ser. NSDI’09.Berkeley, CA, USA: USENIX Association, 2009, pp. 123–136.
[32] S. C. Nelson, G. Bhanage, and D. Raychaudhuri, “GSTAR: generalizedstorage-aware routing for mobilityfirst in the future mobile internet,” inProceedings of the sixth international workshop on MobiArch. ACM,2011, pp. 19–24.
[33] A. Tizghadam and A. Leon-Garcia, “Application platform for smart trans-portation,” in Future Access Enablers for Ubiquitous and Intelligent In-frastructures, ser. Lecture Notes of the Institute for Computer Sciences,Social Informatics and Telecommunications Engineering, V. Atanasovskiand A. Leon-Garcia, Eds. Springer International Publishing, 2015, vol.159, pp. 26–32.
[35] Smart Application on Virtual Infrastructure. [Online]. Available:http://www.savinetwork.ca/
[36] A. Carzaniga, M. Papalini, and A. L. Wolf, “Content-Based Publish/Sub-scribe Networking and Information-Centric Networking,” in Proceedingsof the ACM SIGCOMM Workshop on Information-centric Networking,ser. ICN ’11. New York, NY, USA: ACM, 2011, pp. 56–61.
[37] J. Chen, M. Arumaithurai, L. Jiao, X. Fu, and K. Ramakrishnan,“COPSS: An Efficient Content Oriented Publish/Subscribe System,” inArchitectures for Networking and Communications Systems (ANCS),2011 Seventh ACM/IEEE Symposium on, Oct 2011, pp. 99–110.
[38] H.-A. Jacobsen, “Publish/Subscribe,” in Encyclopedia of Database Sys-tems. Springer, 2009, pp. 2208–2211.
[39] ——, “Content-based Publish/Subscribe,” in Encyclopedia of DatabaseSystems. Springer, 2009, pp. 464–466.
[40] R. Baldoni, M. Contenti, and A. Virgillito, “The evolution of publish/sub-scribe communication systems,” in Future directions in distributed com-puting. Springer, 2003, pp. 137–141.
[41] G. Chockler, R. Melamed, Y. Tock, and R. Vitenberg, “Constructingscalable overlays for pub-sub with many topics,” in Proceedings of thetwenty-sixth annual ACM symposium on Principles of distributed com-puting. ACM, 2007, pp. 109–118.
[42] E. Fidler, H.-A. Jacobsen, G. Li, and S. Mankovski, “The PADRES Dis-tributed Publish/Subscribe System,” in FIW, 2005, pp. 12–30.
[43] Y. Zhang, A. Afanasyev, J. Burke, and L. Zhang, “A Survey of MobilitySupport in Named Data Networking,” in Proceedings of the third Work-shop on Name-Oriented Mobility: Architecture, Algorithms and Applica-tions (NOM’2016).
[44] “Cisco Visual Networking Index: The Zettabyte Era-Trends and Analy-sis,” Cisco, Tech. Rep., 2013.
[45] “Cisco Visual Networking Index: Global Mobile Data Traffic ForecastUpdate, 2013-2018,” Cisco, Tech. Rep., 2013.
[46] M. Rabinovich and O. Spatscheck, Web caching and replication. Addison-Wesley Reading, 2002.
[47] A.-M. K. Pathan, “Utility-oriented internetworking of content deliverynetworks,” Ph.D. dissertation, The University of Melbourne, 2009.
[48] B. Cain, A. Barbir, R. Nair, and O. Spatscheck, “Known Content Network(CN) Request-Routing Mechanisms,” 2003.
[49] J. Pang, A. Akella, A. Shaikh, B. Krishnamurthy, and S. Seshan, “On theresponsiveness of DNS-based network control,” in Proceedings of the 4thACM SIGCOMM conference on Internet measurement. ACM, 2004, pp.21–26.
[50] Netflix Open Connect Content Delivery Network. [Online]. Available:https://openconnect.itp.netflix.com/
[51] D. Rayburn. (2010) An Overview Of Transparent Caching and Its RoleIn The CDN Market. [Online]. Available: http://blog.streamingmedia.com/2010/10/an-overview-of-transparent-caching.html
[52] G. Tyson, S. Kaune, S. Miles, Y. El-khatib, A. Mauthe, and A. Taweel,“A trace-driven analysis of caching in content-centric networks,” in Com-puter Communications and Networks (ICCCN), 2012 21st InternationalConference on, July 2012, pp. 1–7.
[53] P. Agyapong and M. Sirbu, “Economic incentives in information- centricnetworking: implications for protocol design and public policy,” Commu-nications Magazine, IEEE, vol. 50, no. 12, pp. 18–26, December 2012.
[54] G. Carofiglio, M. Gallo, L. Muscariello, and D. Perino, “Modeling datatransfer in content-centric networking,” in Teletraffic Congress (ITC),2011 23rd International, Sept 2011, pp. 111–118.
[55] Y. Wang, Z. Li, G. Tyson, S. Uhlig, and G. Xie, “Optimal cache allocationfor content-centric networking,” in Network Protocols (ICNP), 2013 21stIEEE International Conference on, Oct 2013, pp. 1–10.
[62] A. Tizghadam and A. Leon-Garcia, “Betweenness centrality and resistancedistance in communication networks,” Network, IEEE, vol. 24, no. 6, pp.10–16, November 2010.
[63] E. M. Yeh, T. Ho, M. Burd, Y. Cui, and D. Leong, “Vip: A framework forjoint dynamic forwarding and caching in named data networks,” CoRR,vol. abs/1310.5569, 2013.
[64] R. Mahajan, N. Spring, D. Wetherall, and T. Anderson, “Inferring linkweights using end-to-end measurements,” in Proceedings of the 2Nd ACMSIGCOMM Workshop on Internet Measurment, ser. IMW ’02. New York,NY, USA: ACM, 2002, pp. 231–236.
[65] D. Applegate and E. Cohen, “Making intra-domain routing robust tochanging and uncertain traffic demands: Understanding fundamentaltradeoffs,” in Proceedings of the 2003 Conference on Applications, Tech-nologies, Architectures, and Protocols for Computer Communications, ser.SIGCOMM ’03. New York, NY, USA: ACM, 2003, pp. 313–324.
[66] A. Tizghadam and A. Leon-Garcia, “Robust network planning in nonuni-form traffic scenarios,” Computer Communications, vol. 34, no. 12, pp.1436 – 1449, 2011.