P2P over Mobile Ad-Hoc Networks
Antonio Jose Ribeiro da Silva Marques
Dissertacao para obtencao do Grau de Mestre emEngenharia de Redes de Comunicacoes
JuriPresidente: Prof. Doutor Rui Jorge Morais Tomaz ValadasOrientador: Prof. Doutor Fernando Henrique Corte Real Mira da SilvaCo-Orientador: Prof. Doutor Rui Manuel Rodrigues RochaVogais: Prof. Doutor Luıs Filipe Lourenco Bernardo
Novembro de 2009
Acknowledgments
I would like to thank Professor Fernando Mira da Silva and Professor Rui Rocha for their invaluable
guidance and supervision in the course of the long and difficult process of writing this thesis. Their
considerable knowledge and experience provided me with the tools to pursue my own research interests.
I also extend my thanks to my colleague Carlos Andre for helping in setting up the testbed and in defining
the remote monitoring functionality of the proposed middleware. I would like to thank my parents for their
constant support and encouragement as well as my friends for their patience and understanding during
these past few months.
Abstract
Peer-to-peer (P2P) networks and mobile ad-hoc networks (MANET) share many similarities, such as
decentralisation, dynamic topology and auto-configuration, such that all participants must collaborate to
ensure the correct operation of the network. Furthermore, both types of networks share a common goal:
the discovery of nodes, according to specific criteria, along with the routing of messages between such
nodes. The P2P paradigm is therefore well suited for content distribution, collaboration and communi-
cation applications in a MANET. Despite this, existing P2P systems, designed for the Internet, do not
exploit the synergies between P2P networks and MANETs since the P2P overlay network is unaware
of the topology of the underlying physical network. This can lead to inefficient paths between neigh-
bours in the overlay and unnecessary network overhead due to the duplicate functionality implemented
at different layers in the network protocol stack.
This thesis proposes a P2P middleware solution to facilitate the development of P2P applications
optimised for use in MANETs. To achieve this, the process of searching for P2P application content
is combined with the route discovery process of the MANET routing protocol, Hybrid Wireless Mesh
Protocol (HWMP). When multiple sources are available for the same content, the middleware can choose
a peer based on the cost of the path to that peer, as given by the routing protocol. An implementation of
the proposed middleware is shown to improve performance and reduce the network resource use when
compared to a traditional P2P system layered on top of a MANET.
Keywords
P2P, mobile ad-hoc networks, HWMP, middleware
iii
Resumo
As redes peer-to-peer (P2P) e as redes moveis ad-hoc (MANET) apresentam um conjunto de
semelhancas, tais como descentralizacao, topologia dinamica e auto-configuracao, sendo que os par-
ticipantes nas mesmas necessitam de colaborar para garantir o seu correcto funcionamento. Ambos
os tipos de rede partilham um objectivo comum: a descoberta de nos, de acordo com um determi-
nado criterio, e o encaminhamento de pacotes entre nos da rede. O paradigma P2P surge entao como
uma solucao atraente para a construcao de aplicacoes de distribuicao de conteudos, colaboracao e
communicacao em MANETs. Apesar destas semelhancas, os sistemas P2P actuais, pensados para
utilizacao na Internet, nao exploram as sinergias existentes entre as redes P2P e as MANETs dado que
a rede P2P estabelecida nao tem em conta a topologia da rede fısica sobre a qual esta a operar. Tal
facto pode levar ao estabelecimento de caminhos ineficientes entre vizinhos na rede P2P e a geracao
de trafego excedentario devido a duplicacao de funcionalidades entre as varias camadas de rede.
Esta tese propoe um middleware P2P cujo objectivo principal consiste em facilitar o desenvolvi-
mento de aplicacoes P2P optimizadas para MANETs. Esta optimizacao foi conseguida combinando o
processo de pesquisa de conteudo aplicacional com o processo de descoberta de caminhos por parte
do protocolo de encaminhamento da MANET, o Hybrid Wireless Mesh Protocol (HWMP). Os testes rea-
lizados ao middleware demonstraram um melhor desempenho e uma reducao no consumo de recursos
de rede face a um sistema P2P tradicional a operar sobre uma MANET.
Palavras Chave
P2P, redes moveis ad-hoc, HWMP, middleware
iv
Contents
1 Introduction 2
1.1 Motivation and Goals . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
1.2 Structure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
1.3 Contributions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
2 State of the Art 6
2.1 Peer-to-Peer (P2P) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
2.1.1 Peer-to-Peer Overlay Networks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
2.1.2 Unstructured Overlay Networks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
2.1.3 Structured Overlay Networks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14
2.1.3.A Chord . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
2.1.3.B CAN . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16
2.1.3.C Tapestry . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
2.2 Mobile Ad-hoc Networks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18
2.2.1 Routing Protocols . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18
2.2.2 IEEE 802.11s . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20
2.3 Peer-to-Peer Systems for Mobile Ad-hoc Networks . . . . . . . . . . . . . . . . . . . . . . 22
2.3.1 Unstructured Approaches . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24
2.3.2 Structured Approaches . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26
3 A Peer-to-Peer Middleware for Mobile Ad-hoc Networks 28
3.1 Requirements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29
3.2 Network Architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30
3.3 Peer Architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35
3.3.1 Extending HWMP . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38
3.3.2 P2P-HWMP . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41
3.3.3 Storage Service . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43
3.3.4 Query Routing Service . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45
3.3.5 Service Registration Service . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49
v
3.3.6 Peer Selection Service . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50
3.3.7 Peer Management Service . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51
3.3.8 Middleware Communication Service . . . . . . . . . . . . . . . . . . . . . . . . . . 51
4 Implementation 53
4.1 Extending HWMP . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55
4.2 P2P-HWMP . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 56
4.3 Middleware Daemon . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 58
4.4 API . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62
5 Testing and Evaluation 64
5.1 Testbed . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 65
5.2 Test Metrics and Scenarios . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 66
5.3 Experimental Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 68
6 Conclusion 76
A Middleware Search Message Formats 84
B Middleware SNMP MIB Specification 89
C Middleware API 92
D P2P-HWMP Protocol 96
vi
List of Figures
3.1 Comparison between unstructured P2P search and reactive MANET route discovery . . . 31
3.2 Searching for documents in a MANET by leveraging HWMP route discovery . . . . . . . . 33
3.3 Source peer selection using HWMP path metrics . . . . . . . . . . . . . . . . . . . . . . . 34
3.4 Peer architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36
3.5 Interactions between middleware components at the requesting peer . . . . . . . . . . . . 37
3.6 Interactions between middleware components at the receiving peer . . . . . . . . . . . . . 38
3.7 IEEE 802.11 management frame containing a middleware search request . . . . . . . . . 39
3.8 HWMP vendor registry . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40
3.9 Interactions performed by P2P-HWMP for a received PREQ frame . . . . . . . . . . . . . 42
3.10 Example of a document used by a P2P social networking application . . . . . . . . . . . . 43
3.11 Example of the types of search queries supported by the storage service . . . . . . . . . 44
3.12 Example of a file-backed document . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45
4.1 Peer software stack . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54
4.2 Middleware daemon service dependencies . . . . . . . . . . . . . . . . . . . . . . . . . . 60
4.3 Resource URLs provided by the peer communication interface . . . . . . . . . . . . . . . 61
5.1 Testbed network topology . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 66
5.2 Query forwarding delay for increasing query load . . . . . . . . . . . . . . . . . . . . . . . 69
5.3 Query processing time . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 70
5.4 Query success rate for an increasing query load . . . . . . . . . . . . . . . . . . . . . . . 71
5.5 Overall traffic (bytes) generated for a single query . . . . . . . . . . . . . . . . . . . . . . 72
5.6 Overall traffic (messages) generated for a single query . . . . . . . . . . . . . . . . . . . . 73
5.7 Broadcast traffic (messages) generated for a single query . . . . . . . . . . . . . . . . . . 74
vii
List of Tables
2.1 Classification of peer-to-peer systems and infrastructures . . . . . . . . . . . . . . . . . . 11
5.1 Testbed node configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 65
viii
List of Acronyms
AODV Ad hoc On-Demand Distance Vector
API Application Programming Interface
ARP Address Resolution Protocol
DHT Distributed Hash Table
GUID Globally Unique Identifier
HTTP Hypertext Transfer Protocol
HWMP Hybrid Wireless Mesh Protocol
IP Internet Protocol
IPC Inter-process communication
JSON Javascript Object Notation
LAN Local Area Network
LKM Loadable Kernel Module
MAC Medium Access Control
MANET Mobile Ad-hoc Network
MIB Management Information Base
MTU Maximum Transmission Unit
NDP Neighbor Discovery Protocol
NMS Network Management System
OID Object Identifier
OUI Organisationally Unique Identifier
ix
P2P Peer-to-Peer
PERR Path Error
PID Process Identifier
PREP Path Reply
PREQ Path Request
REST Representational State Transfer
SNMP Simple Network Management Protocol
TCP Transmission Control Protocol
TTL Time-to-Live
UDP User Datagram Protocol
URI Uniform Resource Identifier
URL Uniform Resource Locator
WMN Wireless Mesh Network
XML Extensible Markup Language
XML-RPC Extensible Markup Language - Remote Procedure Call
1
1Introduction
Contents
1.1 Motivation and Goals . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
1.2 Structure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
1.3 Contributions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
2
The rapid growth of wireless communication and mobile computing technologies have led to the
appearance of a new networking concept — Mobile Ad-hoc Networks (MANETs). These networks
comprise a set of autonomous mobile nodes which are not dependent on existing fixed communication
infrastructure to operate. Nodes communicate using wireless links and rely on intermediate nodes to
forward their messages beyond a node’s radio coverage. MANETs, unlike traditional communications
networks, are fully decentralised and have a highly dynamic topology.
In many situations, users cannot rely on an existing network infrastructure: it may be unreliable, too
expensive or simply inexistent. In these cases MANETs may be the only solution. Various applications
for these types of networks have been identified [1]: spontaneous meetings and unplanned events,
where planning and setting up an infrastructure would be too time consuming; disaster relief scenarios
during which existing infrastructure may be damaged and unreliable; military battlefield communications;
sparsely populated remote areas that lack any existing infrastructure or where existing infrastructure is
too expensive.
Another emerging network concept is that of Peer-to-Peer (P2P) networks which allow direct sharing
of resources (CPU, bandwidth, storage) among a large number of users in a decentralised manner [2].
These networks can serve many purposes, from digital content distribution, popularised by applications
such as Napster, Gnutella and BitTorrent, to Internet telephony in, for example, Skype. Unlike the client-
server model, where a limited number of servers provide services and content to a large number of
clients, P2P networks emphasise direct communication among peers acting as both clients and servers.
In pure P2P networks, such as Gnutella [3], all peers perform the same functions and are capable
of sending and replying to request messages sent by any other peer. No central element is used to
manage and coordinate network interactions. Due to the highly transient population of P2P networks,
their network topology is very dynamic, with peers constantly joining and leaving the network. Current
P2P architectures are mainly designed for the traditional fixed and wired infrastructure and can operate
on a global scale using the Internet.
The synergies between these two networking concepts provide an interesting research topic [4].
Both MANETs and P2P networks follow a P2P paradigm characterised by the lack of a central node or
peer acting as a managing server, all participants having therefore to collaborate in order for the whole
system to work. A key issue in both networks is the process of discovering the requested data or route
efficiently in a decentralised manner. MANETs and P2P networks also have a highly dynamic topology:
nodes are constantly moving in and out of radio coverage of neighbouring nodes in the former and peers
are continuously joining or leaving the network in the latter.
Although similar, the integration of both these concepts presents some important challenges. P2P
networks aim to provide resource sharing on top of an existing communication infrastructure, such as the
Internet, where connectivity is not an issue, by forming an overlay network and maintaining connections
3
among its peers. MANETs focus on providing multi-hop wireless connectivity where no infrastructure
exists or is inadequate. P2P application networks operate at the application layer of the OSI reference
model while MANETs operate at the network layer or data link layer. Routing functionality is therefore
present in both layers. On the one hand, application layer routing is used by search queries to find a
peer containing the desired information within the P2P overlay network. On the other hand, network
layer or data link layer routing is used to discover the route to any given node. Another difference lies in
the fact that nodes in a MANET are normally mobile, constrained in computational power, energy and
communication bandwidth, which is usually not an issue for peers connected via wired infrastructure to
Internet wide P2P applications.
These differences give rise to an important challenge in integrating both concepts — existing P2P
systems form an overlay network that is completely unaware and independent from the underlying phys-
ical topology. Logical proximity differs from physical proximity as neighbours in the P2P overlay network
may in fact be located on opposite sides of the planet. This may lead to extremely long routes and unnec-
essary traffic, while not particularly relevant for the high-bandwidth Internet, presents a larger problem
for MANETs due to their lower bandwidth and reliability.
1.1 Motivation and Goals
Consider the increasing push towards using computing technologies and mobile communications in
developing countries and in remote communities where Internet access is intermittent, expensive and/or
unreliable. A prime example of this scenario is the One Laptop Per Child [5] project which envisions the
use of networked laptops in aiding education in the developing world. These cases, where the ability
to access the Internet is severely hindered, stress the importance of decentralised communication and
collaboration tools and the use of MANETs. These can be used to allow villagers to communicate or
humanitarian aid workers to exchange documents. Given the similarities between P2P systems and
MANETs, it is clear that the P2P paradigm is particularly suited to enable content distribution, commu-
nication and collaboration applications in this environment.
The goal of this thesis consists in studying the common issues faced by traditional P2P applications
operating on top of a MANET and proposing a P2P middleware which is optimised for use in a MANET
environment. The proposed middleware should enable P2P application developers to quickly develop a
wide range of applications while leveraging the unique characteristics of these types of networks.
An emerging IEEE network standard, IEEE 802.11s, a mesh networking amendment to the current
IEEE 802.11 standard, will be used as a basis for the construction of the MANET used in the proposed
solution. Although still a draft, IEEE 802.11s, promises to become the de facto standard for the creation
of multi-hop wireless mesh networks and MANETs.
4
1.2 Structure
This text is structured according to the following chapters. Chapter 2, presents a brief overview of
the state of the art of existing P2P systems, both structured and unstructured, and MANETs, namely
the available MANET routing protocols, together with existing research on optimising P2P systems for
use in MANETs. The current draft of the IEEE 802.11s standard is also described along with its default
routing protocol.
Chapter 3 provides a description of the proposed P2P middleware which is tailored specifically for use
in a MANET scenario. This chapter starts with the definition of the requirements which guided the design
of the middleware as well as the chosen architecture, including other solutions which were considered
during the design process. A detailed description of the functionality provided by each middleware
component follows.
Chapter 4 describes the implementation of the proposed middleware for the GNU/Linux operating
system including the selected technologies and main implementation decisions.
Chapter 5 is dedicated to validating and evaluating the performance of the middleware implementa-
tion. This chapter describes the testbed, test scenarios and methodology employed for both functional
and performance tests. To conclude, the experimental results obtained using the testbed are presented
and discussed.
Finally, Chapter 6 draws some important conclusions and presents some suggestions for further
improvements to the proposed middleware.
1.3 Contributions
A middleware architecture for supporting the execution and development of P2P applications opti-
mised for MANETs is proposed as part of this thesis. In addition to this text, the middleware architecture
is also described in an article presented at the 6th IEEE Annual Communications Society Conference
on Sensor, Mesh and Ad Hoc Communications and Networks Workshop (SECON 2009) in Rome, Italy
on June 23, 2009 [6]. A fully functioning implementation of the proposed middleware architecture was
developed for the GNU/Linux operating system and validated using a MANET testbed.
5
2State of the Art
Contents
2.1 Peer-to-Peer (P2P) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
2.2 Mobile Ad-hoc Networks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18
2.3 Peer-to-Peer Systems for Mobile Ad-hoc Networks . . . . . . . . . . . . . . . . . . . . 22
6
This chapter provides an overview of the state-of-the-art in the fields of peer-to-peer (P2P) and
mobile ad-hoc networking, as well as an analysis of current research related to their convergence. First
of all, the typical architecture and properties of a P2P network are presented, along with the main design
features of existing P2P systems and their shortcomings. Subsequently, the main aspects and problems
in mobile ad-hoc networks (MANETs) are described, including a brief look at existing routing algorithms.
Finally, the most promising approaches to provide P2P systems tailored for mobile ad-hoc networks are
detailed and compared.
2.1 Peer-to-Peer (P2P)
A distributed network architecture is designated peer-to-peer if it consists of a set of self-organized
and interconnected nodes, forming a topology that enables the sharing of computer resources (content,
network bandwidth, storage capacity, processing power) through direct exchange, without requiring the
intermediation of a central server. A P2P network must provide acceptable connectivity and performance
in the face of failures and a transient population of nodes [7]. Other definitions for the term peer-to-
peer have been proposed [2] with varying degrees of broadness and, as a result, existing systems or
applications are classified as peer-to-peer according to different definitions. For the rest of this chapter,
the terms node and peer are used interchangeably, where appropriate, to refer to the participants in a
P2P network.
One of the defining aspects of a P2P network resides in its ability to share computer resources
through direct exchange between peers, rather than through an intermediary such as a central server or
authority. In spite of this, many existing P2P systems make use of centralised servers, to some extent,
for tasks such as bootstrapping (the process by which a node joins a network), maintaining reputation
ratings and distribution of encryption keys. Other systems, for example Napster go so far as to use
global centralised servers for basic functionality, like indexing and searching, which, although not fitting
the aforementioned definition, are usually considered P2P systems. The lack of centralised coordination
and management of a P2P network requires the active co-operation of all peers with tasks that include
locating content and other peers, routing messages, maintaining connections to neighbouring nodes
and inserting and retrieving content. Another defining aspect is the capacity to deal with, and adapt to,
network instability due to node and connection failures, in addition to the transient population caused by
churn, i.e., nodes constantly joining and leaving the network. Thus, peer-to-peer topologies must pro-
vide fault-tolerant capabilities, adapting themselves to changing network conditions, while maintaining
adequate connectivity and performance.
Peer-to-peer network architectures serve as a basis for a wide range of applications. These appli-
cations can usually be classified in one of the following categories: communication and collaboration,
7
distributed computing, internet service support, database systems and content distribution. For the us-
age scenarios typically envisioned for mobile ad-hoc networks, communication and collaboration as well
as content distribution applications provide the most relevant and interesting services. The former refers
to the infrastructures and systems that enable peers to communicate and collaborate directly, normally
in a real-time fashion, comprehending applications such as instant messaging (e.g. Jabber [8]) and P2P
telephony (e.g. Skype). The latter focuses on systems that target the sharing of digital media or other
types of data among users, from straightforward file-sharing applications (e.g. Napster, Gnutella [3]) to
complex distributed data storage systems (e.g. PAST [9]).
Content distribution systems are the most popular category of P2P applications and the most exten-
sively studied, therefore, these applications shall be used as examples in the following sections in order
to better illustrate the key aspects of P2P networking.
2.1.1 Peer-to-Peer Overlay Networks
As mentioned previously, a P2P system relies on a network of nodes (i.e. peers) and connections
between them for its operation. This application-layer P2P network is formed on top of the underlying
physical network, typically an IP network such as the Internet, and is referred to as an overlay network.
Current P2P overlay networks are implemented independently from, and unaware of, the underlying
physical network topology, whose routing mechanisms are used to establish the virtual links between
nodes in the overlay.
Peer-to-peer overlay networks are characterised by their structure, degree of centralisation and rout-
ing and location mechanisms used to discover information and route messages within the network.
These attributes influence the overlay network’s fault-tolerance and self-organization capabilities, main-
tenance requirements, performance and scalability. Peer-to-peer overlay networks are usually classified
according to their centralisation and their structure.
The centralisation of a P2P overlay network refers to the role played by central entities, if any, in the
coordination of node activities. In [7], three categories for classifying the centralisation of an overlay
network are proposed: purely decentralised, partially centralised and hybrid decentralised.
Purely decentralised networks correspond to the strictest definition of a P2P network in which all
nodes assume the same roles, thus acting as both clients and servers without any central entity coordi-
nating their activities. The term servent has been coined to refer to such nodes, made up of the terms
server and client. Nodes are therefore responsible for initiating requests for information, forwarding
requests on behalf of other nodes and replying to requests directed towards them.
Partially centralised networks are similar to purely decentralised networks with the addition of the
concept of supernodes or superpeers. These nodes assume a more significant role in the network,
functioning as centralised local indexes for a given set of other conventional nodes. Nodes are assigned
8
the role of supernode dynamically by the network according to certain criteria, for example, processing
power and bandwidth. If a supernode fails, its dependant nodes can be assigned to existing supernodes
or to a newly elected supernode.
Hybrid decentralised networks provide a combination of the traditional client-server model and the
P2P model [2]. Peer-to-peer systems based on this type of network architecture rely on a central server
to coordinate interactions between peers. This server is responsible for tasks such as maintaining a list
of peers connected to the network and descriptions of the information shared by each of those peers.
The central server only stores metadata, allowing it to provide search functionality to peers connected
to it, but unlike the client-server model, end-to-end interaction (e.g. file transfer) is accomplished directly
between peers. Hybrid decentralised overlay networks, given the existence of a single point of failure,
the server, are more vulnerable to technical failure.
Another important aspect used in classifying P2P overlay networks is their structure, which refers to
whether the creation of the network is non-deterministic, where links between nodes are established in
a largely ad-hoc manner, or deterministic, where the overlay topology has to follow precise rules. Thus,
overlay networks fall into one of two categories according to their structure: unstructured or structured.
In unstructured overlay networks there is no restriction on which nodes a given peer can select as its
neighbours. Virtual links in the overlay network are therefore established in an ad-hoc manner resulting
in a random overlay topology that does not follow a specific structure. A peer’s routing table, consisting
of its neighbour nodes, requires minimal maintenance, because an unreachable entry can easily be
replaced with any other node. Furthermore, there is no relationship between the placement of an object
or file and the overlay topology. Since the information a peer is searching for can be located anywhere
in the network, and any of its neighbours is equally likely to possess it, query flooding mechanisms
are employed to locate information. A peer starts by sending a query to its neighbours, which in turn
propagate the query to their neighbours, until the desired information is located and a reply is sent back
to the originating peer. A maximum search radius is used to limit the propagation of queries and to
avoid swamping the entire network with queries. More sophisticated and efficient strategies have been
suggested to avoid the flooding behaviour such as the use of random walks [10].
Although query flooding has an undeniable impact on the scalability of the network, unstructured
overlay networks are extremely resilient to the frequent arrival and departure of nodes making them
suitable for highly-transient node populations. Examples of unstructured P2P systems include Napster,
Gnutella [3] [11] and Kazaa [12].
Structured overlay networks appeared in an effort to overcome the scalability issues raised by un-
structured networks. Rather than allowing the overlay network to be created in an ad-hoc manner,
structured overlay networks not only follow a set of specific rules for their formation, but also store ob-
jects at precisely defined locations. This allows the use of efficient and scalable routing algorithms, able
9
to reliably locate a given object in a bounded number of hops, usually O(log(N)), where N is the number
of nodes in the network. Structured overlays provide a mapping between a given object’s identifier and
the location of the object within the network, i.e., the address of the node with the desired object. This
mapping is achieved through the use of a mechanism known as a distributed hash table (DHT), where
each node in the network is assigned a unique identifier and each object stored within the network is
assigned a unique key. Both node identifiers and object keys exist within the same identifier namespace
and a node is responsible for the objects whose keys are closer to its own identifier, among all live nodes
in the network, according to a predefined distance metric (e.g. numerical distance). A peer can then
perform a lookup on a specified key where each node along the path will forward the request to a node
that is progressively closer to the node responsible for that key. This deterministic routing technique is
also known as key-based routing. Prominent examples of structured overlay networks include the P2P
infrastructures Chord [13], CAN [14], Tapestry [15] and Pastry [16], along with P2P systems based on
these infrastructures such as the PAST [9] and Oceanstore [17].
Despite their object lookup efficiency, structured network overlays have two main disadvantages.
The basic lookup interface offered by DHTs only allows for exact-match queries, i.e., queries where the
exact identifier of the desired object is known. This contrasts with the typical rich queries that can be
performed on unstructured networks, for example, through the use of keywords. Some approaches have
been proposed to layer keyword searches on top of DHTs, such as [18], but suffer from issues related
to load balancing, fault tolerance and hotspots caused by popular keywords. This problem represents
an active research area. Another disadvantage is the maintenance requirements incurred by node joins
and departures in the case of highly transient node populations.
A classification of existing P2P systems and infrastructures according to their centralisation and
structure is presented in Table 2.1. In light of the aforementioned taxonomy, some systems can easily
be identified as unsuitable for deployment in the decentralised environment of a MANET. This is the
case of hybrid decentralised P2P systems, which rely on the use of central entities to implement core
functionality. Since nodes in a MANET have limited resources and are more susceptible to failure when
compared to their counterparts in a fixed network infrastructure, it is unreasonable to expect that a given
node should continuously perform the role of a server, serving multiple clients simultaneously. This
node would not only represent a single point of failure, but would also suffer from increased network
and computational load when compared to other nodes. This uneven distribution of resource usage is
clearly at odds with the basic principles of a MANET. Mobile ad-hoc networks are further discussed in
Section 2.2
The following sections will be dedicated to briefly describing the operation and the main design
features of some of the P2P systems listed in Table 2.1, focusing on purely decentralised P2P systems,
given their similarities with MANETs.
10
Purely Partially Hybrid
Decentralised Centralised Decentralised
Unstructured Gnutella (0.4) Gnutella (0.6) Napster
FreeHaven Kazaa BitTorrent (Tracker)
Structured Chord
CAN
Tapestry
Pastry
BitTorrent (DHT)
Oceanstore
PAST
Table 2.1: Classification of peer-to-peer systems and infrastructures
2.1.2 Unstructured Overlay Networks
Gnutella [3] is the most prominent example of a purely decentralised unstructured overlay network
due to its widespread use, open protocol and large scale network topology. It provides a P2P overlay
network with appropriate routing and location mechanisms in order to allow users to share files with
other participants in the network. Given its completely decentralised nature, no central coordination
exists and all nodes perform the same functions, acting as both a client and a server (servent) and
connecting directly to each other.
The Gnutella application layer protocol is used along with the IP routing mechanisms of the under-
lying network to enable communication between peers. The initial versions of the protocol [3] specify
four basic types of messages: ping, pong, query and query hit. The ping message is used to discover
other hosts in the network by requesting that a given node identify itself to the sender. This identifi-
cation, including the IP address and port, is conveyed in a reply message known as pong. The query
message represents a search request and contains the search terms along with parameters specify-
ing quality requirements the replier should satisfy. The response to a query message is conveyed in
a query hit message which contains the identification of the responding node along with the result set.
These message types concern only the act of joining and searching the Gnutella network. File transfer is
done directly between peers, i.e., not through the Gnutella network, using the hypertext transfer protocol
(HTTP).
A node that wishes to connect to the Gnutella network must know the address of at least one other
node that is already connected to the network. This can be done by consulting one of the many well
known GWebCache servers which maintain an updated list of hosts. A client then sends a ping message
to each of its known hosts. Accessible nodes will respond with a pong message, identifying themselves,
11
and propagate the ping request to their neighbours. After obtaining a given number of working host
addresses, the original node can begin issuing search queries. Host addresses that a node learns
through GWebCache, pong replies and successful replies to queries are cached locally, so that future
sessions can be established without consulting a central database.
The search mechanism used in Gnutella reflects the typical approach followed by unstructured over-
lay networks referred in Section 2.1.1. Since there is no relationship between files and the nodes re-
sponsible for storing them, and files can be located anywhere within the network, a non-deterministic
search method is used. The initial versions of the Gnutella protocol, up to and including version 0.4,
specify a pure query flooding mechanism where a requesting node sends its search request, the query
message, to all its neighbours, which in turn forward the message to all their neighbours with the ex-
ception of the node from which they received the message. When the search criteria match one of the
files shared by a given node, a query hit message is sent back along the same path that was followed
by the original query message. Each message contains a time-to-live (TTL) field in its header to avoid
continuously forwarding it and unnecessarily flooding the network. This field is decremented at each hop
until it reaches zero, at which point the message is no longer forwarded. Furthermore, each message
has a unique identifier which allows a node to avoid routing loops and to preserve network bandwidth
by dropping messages whose identifier it has already seen and thus already processed. The identifiers
of both pong and query hit messages are equal to their respective request messages. This allows the
implementation of the reverse path routing previously discussed. Upon receiving a query hit message,
a node will contact the responding host directly and negotiate the transfer of the requested file via HTTP.
Simple query flooding limits the ability for this type of network to scale to an increasing number of
nodes and queries. There is an obvious trade-off between the selected TTL value and the network
traffic generated in addition to the amount of nodes searched. On the one hand, higher TTL values will
result in more nodes being contacted at the expense of increased network traffic, which may swamp the
underlying physical network. On the other hand, reducing the TTL value will reduce the network traffic
but at the expense of less nodes contacted and consequently fewer search results.
To address this problem, the latest version of Gnutella, version 0.6 [11], adopts a two-level hierarchy.
As such, Gnutella should now be classified as a partially centralised P2P system. The main difference
is the introduction of the concept of ultrapeer, a peer that is automatically elected in order to service
a small subset of regular peers, known as leaves, indexing the files contained by each of those peers.
Peers are elected as ultrapeers if they provide higher processing power, network bandwidth or even
higher uptime than other peers. Ultrapeers are responsible for the leaf nodes connected to them. Leaf
nodes will send a list of shared files to their ultrapeers, in order to allow them to index such files. Search
queries are also sent directly to the ultrapeers, which will then proxy the requests on behalf of the leaf
node. Upon receiving a search request, an ultrapeer will forward this request to other ultrapeers using a
12
query flooding method analogous to the one used on the purely decentralised version. An ultrapeer will
only forward a query to one of its leaves if it knows the leaf contains the requested content.
The use of ultrapeers in a partially centralised overlay network brings a few advantages over the
purely decentralised approach. Query latency, network traffic as well as the number of nodes involved
in routing a query are reduced since queries are propagated only among ultrapeers. This is achieved
while still maintaining the robustness and fault-tolerance characteristics of purely decentralised net-
works, since ultrapeers do not constitute single points of failure and can easily be replaced. Another
advantage of this approach is that it exploits the heterogeneity among nodes such as their network
bandwidth, storage capacity and processing power. Ultrapeers bear much of the load of the network,
freeing leaf nodes from costly routing and processing tasks. A more detailed explanation on the opera-
tion of the partially centralised Gnutella network can be found in [11]. Kazaa [12] is another example of
a partially centralised P2P system.
Several solutions have been suggested for improving the performance of search queries in unstruc-
tured overlay networks. These methods focus on reducing query latency, i.e., the time required to find
the specified content, and/or the amount of network traffic generated by search queries. Improving these
metrics thus contributes to increased system performance.
In [10], two alternatives to the standard query flooding approach are suggested: expanding ring
search and random walks. Expanding ring search is proposed as a solution to the problem of selecting
an appropriate TTL value. A node will use successive query floods with an increasing TTL. Starting with
a low TTL value, the node initiates a query flood for the requested content. If the search is unsuccessful,
the node increases its TTL value and tries again, repeating this process until the requested content is
found. Clearly this method presents a performance advantage when used to locate popular content with
higher replication ratios but does not address the issue of message duplication inherent in the flooding
approach. Alternatively, the authors propose the use of multiple parallel random walks, in which each
node will choose a random node from its set of neighbours to forward the query message to. This
message is called a walker. A walk will continue until the requested content is found, the message TTL
is exceeded (this is usually a large value when compared to standard query flooding and is used only to
avoid loops) or the originating node already has enough results (each walker periodically checks back
with the originating node). This method, when combined with proactive replication mechanisms, can
reduce network traffic by an order of magnitude when compared to expanding ring search.
Other more sophisticated search mechanisms are proposed in [19]. These include the use of intel-
ligent forwarding strategies where a node will choose a given neighbour, based on its performance in
past queries, for forwarding its search requests to. Each node maintains a set of simple statistics on its
neighbours allowing it to apply heuristics such as selecting the neighbour through which the most query
results have been received or the neighbour which returns messages with the lowest average hop count.
13
Assuming an intelligent neighbour selection, this method is shown to significantly reduce network traffic
while maintaining the quality of results. Another approach to reduce the number of nodes involved in
query processing and routing relies on the concept of local indices. Each node maintains a local index
of all the data shared by nodes which are located within a fixed number of hops from itself. Thus, a node
which receives a query can process it on behalf of any node within a given radius.
Another approach that relies on a distributed index mechanism is proposed in [20]. The concept
of routing indices is introduced which relies on the existence of an index at each node that provides
an indication of the neighbour that is most likely to be in the “direction” of the requested content rather
than the actual location of such content. The main objective of the routing index is to rank a node’s
neighbours according to their “goodness” for a given query. In the example presented, a node’s routing
index contains the number of documents along each path and the number of documents on each topic
of interest. A node can then select the neighbour whose path contains the highest number of documents
for a particular topic. This method is shown to improve performance by one or two orders of magnitude
over traditional query flooding.
In general, unstructured peer-to-peer systems can be an attractive choice when keyword searching is
desirable, the node population is highly transient, searches focus on content that is sufficiently replicated,
network size is adequate for the search methods used and guaranteed content discovery is not required.
2.1.3 Structured Overlay Networks
Structured overlay networks provide a purely decentralised, efficient, and scalable distributed object
location and routing mechanism. Most of these mechanisms rely on a hash-table interface that serves as
the foundation for a distributed lookup system. This interface places few constraints on the type of data
stored by the system. The only requirements are that the data must be identified by a unique numeric
key and nodes must be willing to cooperate by storing (key, value) pairs for each other. Responsibility
for mapping keys to the nodes that contain their values is distributed among the nodes so as to minimise
the changes required when a node joins or leaves the network.
A DHT must implement a single operation, lookup(key), which allows a node to lookup a given key
and obtain the identity of the node responsible for said key [21]. Consider a distributed file storage
system, a user wishing to store a particular file under a unique name would start by applying a hash
function, such as SHA-1, to the name of the file in order to obtain a numeric key. He or she would then
send the file to the node address that results from calling the function lookup(key) on the file’s key. Later
on, someone wishing to obtain this file, would first obtain its name, apply the same hash function to
obtain the corresponding key, call the lookup(key) function and contact the resulting node address to
obtain the file. This relatively simple example ignores caching and replication strategies, which are also
important in guaranteeing content availability.
14
A DHT based lookup algorithm must address a series of issues for its correct operation. Keys must
be mapped to nodes in a way that balances their distribution evenly among participating nodes. This is
done by mapping nodes and data or data identifiers (e.g. file names) to the same abstract key-space of
n-bit keys, using a standard hash function. A key k is therefore assigned to the node whose identifier
(i.e. key), is closest to k (according to a given metric such as Euclidean distance or numeric distance)
among all nodes within the network overlay. Another issue is the forwarding of lookup queries for a key
to the appropriate node. A node which receives a query for a given key k must be able to forward it to
a node whose identifier is closer to k in order to guarantee that the query will eventually arrive at the
correct node. Thus, each node maintains a small routing table with carefully chosen nodes. In some
systems the query is forwarded to a node whose identifier is larger than the current node but still smaller
than the requested key, while in other systems the query is forwarded to a node whose identifier has a
longer prefix in common with the key than the current node. Finally, routing tables must be constructed
by selecting appropriate nodes. To support routing as described previously, a node must either know
about its sucessor, which is the node with the next largest identifier in the key-space, or a list of nodes
with matching prefixes of different lengths.
This section will provide a brief overview of the design and operation of the most prominent examples
of DHT based peer-to-peer infrastructures: Chord [13], CAN [14] and Tapestry [15]. Although these
infrastructures do not constitute working applications, they provide a powerful abstraction, the hash
table interface, that can be used as a building block for peer-to-peer applications. They provide efficient
and fault tolerant routing and location services through which objects can be efficiently located. Despite
the fact that these infrastructures offer the same abstraction and services, they differ in the algorithms
used for overlay construction and maintenance [21].
2.1.3.A Chord
Chord [13] allocates keys and node identifiers from the same, one-dimensional, identifier space using
a consistent hashing function. This space is organised as a circle and wraps around such that the node
with identifier 0 follows the node with the highest identifier. A node is responsible for a given key k if
its identifier is equal to k or most closely follows k. This node is also known as the successor of k.
Consistent hashing ensures the load is balanced by assigning roughly the same amount of keys to each
node. In order to successfully route a query, a node must maintain a single piece of information, a
pointer to its successor in the identifier circle. A query can then be routed around the circle following
successor pointers until the node responsible for the key is found.
To ensure correct node lookups despite failures, Chord maintains a successor list containing the
addresses of the next r nodes in the identifier circle. This allows incremental progress to be made in
spite of a failed successor node, allowing up to r−1 simultaneous node failures before the node corrects
15
its successor list. Since successive nodes on the identifier circle are likely to be unrelated in the physical
topology, their failures can be considered independent and therefore even with small values of r the
probability of simultaneous failure is negligible.
Routing a query by relying exclusively on the successor list is an inefficient process. In the worst
case, a query will be forwarded by all N nodes in the network. To optimise the routing process, nodes
maintain, in addition to the successor list, a finger table with logN entries. Each entry i of the finger table
corresponds to the successor of the key n + 2i, where n is the current node’s identifier. For that reason,
the finger table maintains the address of a node that is at least half-way around the identifier space, the
address of another node that is at least quarter-way around and so forth. A node that wishes to forward
a query for key k, will select the entry in the finger table with the highest identifier that is less than k.
As a result of halving the distance in identifier space at each hop, Chord requires O(log(N)) time and
messages to resolve lookups.
To join a network, a new node n must ask an existing node in the network to lookup its identifier and
discover its successor. Responsibility for some of the keys assigned to the successor will be transferred,
along with the corresponding data, to the new node and in order to re-establish routing correctness,
both n and its predecessor need to update their successor lists. When n leaves, all its keys must be
transferred to its successor. The number of nodes involved in key reassignments when a node joins and
departs is minimal.
2.1.3.B CAN
Another proposed DHT abstraction is the content addressable network (CAN) [14]. CAN defines a
virtual d-dimensional Cartesian coordinate space to which it maps (key, value) pairs. This space is split
up into a set of hyper-rectangles called zones. Each individual node is responsible for a given zone,
which stores part of the hash table, and is identified by the boundaries of its zone. A key k is thus
mapped deterministically to a point P in the coordinate space and is stored at the node responsible for
the zone that contains P . Each node also maintains a routing table containing the addresses of the
nodes covering adjacent zones, which allows a query to be routed towards the node responsible for the
key.
To retrieve the value corresponding to key k, the same process is used to map k to a point P . If
P does not lie in the zone covered by the requesting node, the query is forwarded along the path that
approximates a straight line in coordinate space from the requesting node to the node storing k. The
lookup time for CAN is O(dN1d ).
In order to join the network, a node uses a bootstrap mechanism to discover a node already on the
network. It then selects a random point P in coordinate space and asks the latter to find the node n
which is responsible for the zone containing P . The new node contacts n, which in turn splits its zone in
16
half and assigns one of the halves to the new node. The new node must then initialise its routing table
using the addresses of n’s neighbours and n itself and advertise the new zone so its new neighbours
can update their routing tables. When a node leaves the network, its zone and corresponding keys
must be handed over to one of its neighbours. A node will periodically inform each neighbour of its
zone coordinates and that of neighbouring zones. This allows neighbours to detect when a node has
failed, through the absence of updates, and initiate a coordinated protocol to allow the neighbour with
the smallest zone to take-over the failed node’s zone.
2.1.3.C Tapestry
Tapestry [15] is based on the lookup mechanisms described in [22], namely the distributed data
structure known as the Plaxton mesh which allows distributed object location and routing using rout-
ing maps of small and constant size. Each node maintains a routing map, known as neighbour map,
consisting of multiple levels. A given level l contains addresses of nodes whose identifiers match the
current node’s identifier up to a certain prefix length of l digits. The number of node addresses stored
for each level is equal to the base of the identifiers used (each entry can contain further backup entries
to increase reliability). Furthermore, each of these entries corresponds to the neighbour closest in the
network whose identifier satisfies the prefix matching condition. As a message is routed, it is forwarded
to neighbour nodes that are progressively closer to the destination. For example, if node with identifier
1872 receives a request for the object with identifier 1833, both identifiers share a common two digit
prefix. The forwarding node must therefore select a node from its neighbour map, such as 1839, whose
identifier shares an additional matching prefix digit with the requested object. A node must therefore
maintain a neighbour map that, for each possible prefix of its own identifier, contains the address of a
node that matches that prefix but differs in the next digit. Tapestry guarantees that, in a stable state,
any node can be reached in logβN overlay hops, where N is the size of the identifier space and β is the
base of the identifiers.
An object with a given identifier o is mapped to a unique live node, its root node nr, which is guar-
anteed to know its location. A server ns, which stores o, will publish the object o by routing a message
toward its root nr. Along the routing path, each node stores a mapping (o, ns), rather than storing the
object itself. In the presence of replicas, each server storing a replica of o will repeat the same operation.
Tapestry then stores the mappings for o’s replicas in sorted order according to the latency measured from
the node holding the mappings. Object lookups are performed by routing a query message towards the
object’s root node nr. Each node along this path will check its mappings for the object, if a mapping
exists for that object, the message is routed directly to the server referred in the mapping, otherwise it
continues in the direction of nr which is guaranteed to know the object’s location. This mechanism is
used to allow nodes to discover object replicas that are close by.
17
The primary advantage in using these structured peer-to-peer systems resides in their object lookup
efficiency. Search performance, in number of hops, is O(logN) for most systems, where N is the network
size, as is the space occupied by the basic routing tables. Unfortunately, efficient operation in a stable
state requires rather costly operations on node join and departure, which is apparent given the afore-
mentioned algorithms. This makes such systems less suitable for highly transient node populations.
Another disadvantage is related to the fact that the above systems only support exact match lookups,
i.e., based on an item’s unique key identifier.
2.2 Mobile Ad-hoc Networks
A mobile ad-hoc network (MANET) is characterised by a set of mobile devices that communicate
among themselves using wireless links. This communication is done in a completely decentralised, self-
organised manner and without requiring any pre-existing fixed communications infrastructure. These
networks are particularly useful in scenarios where the existing infrastructure is unreliable, too expen-
sive, or simply non-existent. This includes unplanned meetings, disaster relief, military battlefield com-
munications, public safety and communications in remote or rural areas. The latter is a particularly
common occurrence in the developing world.
Nodes in a MANET have to rely on each other to forward packets beyond their radio coverage,
forming a multi-hop wireless network. Generally, each of these nodes merges the role of an end-system
and a router. A key aspect that distinguishes MANETs from traditional wired or wireless infrastructures
is that they lack a fixed topology, nodes are mobile and organise themselves arbitrarily. Node mobility
also contributes to the highly dynamic network topology, where nodes are constantly joining and leaving
the radio coverage of other nodes.
2.2.1 Routing Protocols
Routing, given the dynamic nature of MANETs, is one of the most important problems when ensuring
communication between two arbitrary nodes. It is used to determine the path between a source node
and destination node and in forwarding data between them. MANETs exhibit a few important differences
when compared to wired networks that limit the applicability of existing routing algorithms designed for
wired networks. These differences include the possibility of asymmetric links, since each endpoint may
have different antennas and transmit power. Routing information determined for one direction may not be
suitable for the reverse direction. Another issue is interference which can be caused by the broadcast
nature of wireless communications present in MANETs. Nodes may interfere with each other when
transmitting and links tend to be unreliable due to various environmental factors which in turn can lead
to a higher percentage of lost packets. Also, as mentioned previously, the topology is highly dynamic.
18
As nodes move or the environment changes, the set of currently valid routes may change, invalidating
existing routing information and forcing the routing algorithm to recalculate routes. Finally, nodes in a
MANET have limited resources due to their lack of wiring and smaller physical size. Limitations include
processing capacity, memory and battery power, thus, optimising resource usage is a priority.
Mobile ad-hoc networking has attracted much attention from the research community and dozens of
routing protocols have been proposed for MANETs. A detailed analysis of routing protocols for MANETs
can be found in [23]. Rather than describing such algorithms in detail, the rest of this section will highlight
their main design features, advantages and disadvantages. According to [1], MANET routing algorithms
can be classified into three categories: flat ad-hoc routing, hierarchical routing and geographic-position-
assisted routing.
In a flat ad-hoc routing protocol, no node or address hierarchy exists and all nodes perform equivalent
routing roles within the network, operating according to the same routing protocol. This category of
protocols is further subdivided into reactive and proactive routing protocols.
Proactive protocols set up routing tables at each node and keep them updated regardless of the
actual communication needs of the node. In other words, a proactive protocol will maintain routes to
destinations that are not currently used for packets sent by the node. Link-state based algorithms are
commonly used to implement proactive protocols. In this case, routing information is exchanged via
broadcast with neighbouring nodes periodically or as a result of network events. The primary advantage
of proactive protocols is the fast connection setup. When a node needs to transmit a packet to a given
destination node, there is a high probability that a route to that node already exists in the routing table.
Thus, packets do not experience the higher delay that would be caused by an expensive route discovery
process. Furthermore, if changes to the topology are infrequent, a node will have a fairly accurate view
of the current network topology, including the costs (e.g. latency, bandwidth) associated with the paths
it chooses. Unfortunately, their proactive nature represents an unnecessary overhead in lightly loaded
networks, as routing table updates are exchanged regardless of existing communication needs. Given
the mobile nature of nodes and their limited resources, additional traffic will further drain a node’s battery.
It may be difficult to define an adequate trade-off between the frequency of routing table updates, in order
to maintain an accurate view of the current topology, and preservation of already scarce resources.
Prominent examples of proactive protocols for MANETs include DSDV [24] and OLSR [25].
Reactive protocols provide another alternative for flat ad-hoc routing. A reactive protocol avoids
unnecessary maintenance traffic by discovering routes on demand. In other words, when a source node
wishes to send a packet to a given destination node, the underlying reactive routing protocol will initiate
a process to discover the path between both nodes. Discovered routes are cached at each node and
removed from the routing table after a certain period of inactivity. The obvious advantage of reactive
protocols is that they limit maintenance traffic to that which is required by currently used routes. This
19
offers a scalable solution if a node tends to communicate with the same set of nodes and node mobility
is limited, allowing a route to be discovered once and maintained for a long period of time. Additionally,
nodes can better leverage power saving periods in lightly loaded networks given the lack of periodic
route updates. Disadvantages include the time required to setup a route before a packet can be sent
which will impact the latency of the first packet in a connection. This can be problematic for interactive
applications. The overhead incurred by route discovery may also make reactive protocols unsuitable if
a node must communicate with an ever changing set of nodes. AODV [26] and DSR [27] are the most
representative examples of reactive protocols.
Rather than assume that node movement is independent, hierarchical ad-hoc routing protocols group
nodes into clusters of nodes that follow the same movement pattern. These protocols are based on the
idea that members of a group tend to move together and therefore a node will most likely remain within
the same cluster. This allows a node to move freely within its cluster and only inform other cluster
members, abstracting node movement within a cluster so that members of other clusters only need to
know how to communicate with one of its members. One or more nodes are chosen as clusterheads,
these gateway nodes are responsible for routing packets to and from other clusters. Clusters can then
be organised into a hierarchy. Advantages of this approach, alongside scalability, include the ability
to reduce routing table sizes, to shield nodes within a cluster from mobility in other clusters and to
use different routing protocols, with possibly different update frequencies, inside and outside clusters.
Disadvantages include the difficulty in maintaining the structure of clusters in the face of high mobility
(which has a particularly adverse effect when clusterheads change groups), the possible bottleneck
presented by gateway nodes (these nodes also suffer greater resource usage) and the use of sub-
optimal paths. HSR [28] is an example of a hierarchical routing protocol.
Finally, geographic-position-assisted ad-hoc routing protocols make use of a node’s knowledge re-
garding its geographic position to improve routing performance. This position can be obtained through
various location systems, although the most common is the global positioning system (GPS). An exam-
ple of this category of protocols is GPSR [29], where a node periodically exchanges position information
with its neighbours. Routing is then performed by greedily forwarding data packets to the neighbour that
is geographically closest to the destination node.
2.2.2 IEEE 802.11s
IEEE 802.11s is a proposed wireless mesh networking amendment to the IEEE 802.11 standard.
While currently still a draft and the focus of ongoing work by IEEE 802.11 task group s, IEEE 802.11s
specifies a set of mechanisms that enable wireless devices to form ad-hoc networks and communicate
among themselves. It has been designed with flexibility in mind and applicability to a variety of different
networking scenarios. MANETs and wireless mesh networks (WMN) share many similarities [30]: they
20
are both self-organising, decentralised, wireless multi-hop networks and must deal with dynamic network
topologies caused by node mobility and changing environment. WMNs can generally be classified into
two categories: back-haul meshes and client meshes. While back-haul meshes provide a wireless
access network based on an infrastructure of meshed fixed wireless access points, client meshes consist
exclusively of end-user devices (e.g. laptops) which are usually mobile. Client meshes are therefore
equivalent to MANETs providing a flat hierarchy of mobile nodes with identical functionality.
The flexibility of IEEE 802.11s allows it to be applied to either mesh networking scenario and does
not place any constraint on the types of devices used. These can be either fixed infrastructure devices
or mobile devices. It specifies a set of medium access control (MAC) layer enhancements to the IEEE
802.11 standard, focusing on areas such as multi-hop wireless routing, radio-aware metrics, power
management and security. The routing and forwarding mechanisms are of particular interest to this
thesis and will be described in further detail in this section.
The architecture of an IEEE 802.11s network consists of nodes termed mesh points (MPs). Mesh
points are IEEE 802.11 devices that provide basic forwarding functionality within the mesh and there-
fore participate in the routing protocol defined in IEEE 802.11s. These mesh points establish links,
called mesh links, among themselves to form the multi-hop mesh topology. Mesh points may be mobile
nodes or fixed nodes that are part of the infrastructure. Some mesh points may provide access point
functionality to traditional IEEE 802.11 clients. These are called mesh access points (MAPs). Another
type of mesh points, which provide additional functionality, are the mesh portals (MPPs). These act as
gateways, connecting the mesh network to other IEEE 802 networks.
A central component of IEEE 802.11s is its default routing protocol — hybrid wireless mesh protocol
(HWMP). Any device that claims to be compliant with IEEE 802.11s must implement HWMP, although
other protocols may be used in addition to it. HWMP is a hybrid routing protocol that contains a reactive
and a proactive component.
The reactive component is based on a modified version of AODV [26], implemented at the MAC layer
and with radio-aware link metrics. Since it is an on-demand algorithm, path discovery is only performed
for the destination nodes the source node communicates with and discovered paths are cached until they
are no longer needed. To ensure that recent path information takes precedence over older information,
sequence numbers are used. When a source mesh point wishes to discover a path to a destination mesh
point for which it does not have a route in its routing table, it broadcasts a path request (PREQ) message
across the mesh network. A mesh point that receives a PREQ will update its routing information and add
a temporary back pointer to the mesh point it received the message from. It can reply with a path reply
(PREP) message if it is the destination of the request or if it has a more recent path to the destination
mesh point than that which is identified in the PREQ. In this case, the PREP is unicast back to the
source mesh point, otherwise the PREQ is rebroadcast. PREQs that have already been seen by a mesh
21
point are identified and discarded. As the PREP travels backward towards the source mesh point, each
intermediate mesh point adds a forward pointer to the destination. Upon receiving the PREP, the source
can start using the discovered path.
The proactive component is provided as an extension that allows the creation of tree structures based
on specific mesh points, the root mesh points, that usually coincide with MPPs. These root mesh points
will periodically broadcast proactive PREQ messages to the network to maintain a tree of paths leading
to themselves. This allows efficient routing when communicating with nodes outside the mesh.
2.3 Peer-to-Peer Systems for Mobile Ad-hoc Networks
In spite of the popularity of both P2P technologies and MANETs, researchers have only recently
begun to explore the synergies that exist between them. Several common features can be identified
between pure P2P networks and MANETs [4] [31] [32]. Neither a P2P network nor a MANET have any
central peer or node responsible for facilitating the exchange of information or coordinating network ac-
tivity. All nodes or peers must collaborate among themselves, forwarding information on behalf of others
in the network, in order to guarantee basic system operation. Furthermore, all participants assume the
same roles in the network: initiating, forwarding and servicing requests. Another common aspect of both
networks is the primary problem they attempt to solve, that of locating the desired information or route
in an efficient manner, i.e., application and network layer routing respectively. Both networks also have
a highly dynamic topology due to nodes joining and leaving the network, and in the case of a MANET,
the mobility of nodes. Finally, broadcasting is used in both systems to some extent to exchange data or
routing information with neighbouring peers.
Significant differences also exist which present additional challenges when trying to integrate both
approaches. P2P systems rely on overlay networks, implemented at the application layer, while MANETs
are only concerned with the lower layers of the OSI model. Due to this clear separation of layers, current
P2P overlay networks are largely unaware of the underlying physical network topology and assume
the existence of efficient underlying IP routing mechanisms. P2P systems are usually designed for
the Internet where peers are often static, have plentiful resources (e.g. network bandwidth, processing
power, storage capacity) and network links are highly reliable. None of these assumptions hold for
nodes participating in MANETs. Furthermore, overlay broadcasting in a P2P network is done by using
multiple unicast messages but in MANETs any data transmission is inherently broadcast due to the
shared physical medium. Another issue that stems from the mismatch between physical and overlay
topology is the fact that neighbours, nodes one hop apart, in the overlay can be many hops apart in
the underlying physical network. Although not particularly problematic for Internet scale P2P systems,
where the cost of reaching any node is assumed to be similar and high bandwidth links compensate
22
for sub-optimal paths, communicating with distant nodes across multiple hops in a MANET will lead to
much lower throughput and higher network usage.
The aforementioned features of P2P systems and MANETs, including their similarities and differ-
ences, justify further investigation in adapting current P2P technologies and solutions, developed for the
wired Internet, to the unique operating environment of mobile ad-hoc networks. The rest of this section
provides an overview of existing studies, proposals and solutions that discuss the use of P2P systems
for mobile ad-hoc networks.
The use of an unstructured P2P application, a modified version of Gnutella, over three different ad-
hoc routing protocols (DSR, AODV and DSDV) is considered in [32]. The results show that none of
the ad-hoc routing protocols provide a clear performance advantage over the others, with each protocol
performing well under some scenarios for certain metrics and suffering from drawbacks in others. The
characteristics of the application, network and nodes in terms of workload, number of peers, mobility
and network density are shown to have a large influence over the performance of the underlying ad-hoc
routing protocol. In addition, the implemented P2P application makes no attempt to adapt the overlay
topology to the underlying physical topology or to integrate both routing protocols. Further research by
the same authors [33] compares a similar unstructured P2P application with an structured P2P system,
an implementation of Chord, again without any specific optimisations for the MANET scenario. The
results presented show that the unstructured P2P system is more resilient to a transient node population
than the structured system. This comes at the cost of increased network bandwidth usage, energy
consumption and delay. Structured P2P systems on the other hand use less resources but are better
suited for static topologies.
A theoretical comparison of routing approaches that combine P2P search protocols and MANET
routing protocols is presented in [4]. Five different routing approaches are suggested (broadcast over
broadcast, broadcast, DHT over broadcast, DHT over DHT and DHT) and compared in terms of their
routing complexity, scalability, implementation complexity, maintenance complexity, energy efficiency
and use of optimal paths. The authors conclude that the use of cross-layer routing approaches, where
the virtual P2P overlay is eliminated and application-layer searching is integrated with network-layer
routing, provide a significant performance advantage over non-integrated approaches. This applies to
both broadcast based and DHT based routing approaches. The former is an ideal candidate for MANETs
of small size and the latter can provide scalable and efficient search in larger MANETs, so long as the
routing and neighbour tables can be appropriately maintained. This analysis is unfortunately limited to
route discovery and does not refer how routing tables are constructed and maintained when using DHT
based approaches, nor does it refer practical issues such as node mobility and failure.
A comparison of several cross-layer approaches, which try to improve the performance of P2P sys-
tems over MANETs, is provided in [31]. Since the simple application of existing P2P systems over
23
MANETs can lead to poor performance and duplication of routing functionality at both the application
and network layer, as described above, several systems have been proposed that follow cross-layer ap-
proaches, i.e., where application layer routing protocols and network routing protocols are integrated or
interact with each other.
A brief overview of existing P2P solutions designed for use in MANETs is presented in the following
sections. Both unstructured and structured, DHT-based, approaches are considered.
2.3.1 Unstructured Approaches
7DS [34] is a peer-to-peer data sharing and dissemination system for nodes that may have inter-
mittent connectivity to the Internet. It is focused on urban environments with a high spatial locality for
information and enables a user’s information needs to be serviced by nearby nodes when not connected
to the Internet. It can operate in a pre-fetch mode, anticipating those needs, or on-demand, searching
for information among peers only when the device fails to access data directly via the Internet. Col-
laboration can be based on data sharing, forwarding messages for other nodes and caching popular
information such as web pages. 7DS does not assume that all nodes are collaborative or have the same
capabilities and allows the use of fixed infrastructure devices, the 7DS servers. These servers provide
services to non-collaborative clients which clearly contrasts with the MANET paradigm. Furthermore,
the described system relies only on single-hop multicast to obtain data items from nodes that are within
wireless coverage. This usage scenario assumes high node mobility in disseminating information rather
than relying on any routing mechanisms.
Proem [35] provides a mobile P2P platform aimed at facilitating the development and deployment of
collaborative mobile P2P applications in MANETs. It is characterised by its ability to adapt to a rapidly
changing environment, universal applicability to a wide variety of P2P application requirements (from file
sharing to instant messaging), platform independence using web standards and protocol extensibility. It
defines the concept of data space, a collection of data managed co-operatively by a set of peers, and
the concept of communities, providing a way to group entities (e.g. peers, data spaces) and control
access through the use of memberships. Proem defines four protocols, an XML (Extensible Markup
Language) based message oriented transport protocol, a presence protocol to advertise devices and
services, a data protocol for data sharing and synchronisation and a community protocol to manage
community memberships. The Proem platform only defines application layer protocols and does not
provide integration with the services offered by the underlying MANET, namely routing, leading to a
mismatch between virtual an physical topologies.
An unstructured P2P file-sharing system, specifically tailored for MANETs, is presented in [36].
ORION (Optimised Routing Independent Overlay Network) consists of a search algorithm, responsi-
ble for locating files and establishing routes, and a file transfer protocol. An application-layer overlay
24
network is formed by routes which are discovered on-demand by the search algorithm; ORION thus
combines application-layer query processing with the process of discovering network routes. These
routes are maintained only as long as necessary and closely match the underlying network topology.
Redundant paths to each file are also maintained. The ORION search algorithm provides a significant
reduction in control traffic and increases search accuracy when compared to Gnutella layered on top of
DSR. In addition to the search algorithm, the file transfer protocol offers lower overhead and increased
probability of successful file transfers when compared to TCP. Although ORION uses many structures
and mechanisms that are analogous to those used in reactive routing protocols for MANETs, it does
not integrated with, and does not require, any MANET routing protocol to operate. Search queries are
sent using link-layer broadcast messages and replies are sent along the reverse path, similar to AODV.
As a result, ORION does not exploit any synergies with MANET routing protocols that may already be
present in the protocol stack.
A performance analysis of Gnutella when faced with typical MANET conditions such as node mo-
bility, network partitions and node replacements, is described in [37]. Both AODV and OLSR are used
as underlying MANET routing protocols for this evaluation. Despite the fact that Gnutella provides the
necessary capabilities to enable P2P data sharing in an ad-hoc environment, several shortcomings are
identified. When faced with increasing node mobility, Gnutella is unable to satisfy minimum connec-
tivity requirements for the overlay network. Furthermore, network partitions generate traffic bursts due
to the reconfiguration of network topologies, illustrating the negative impact of highly dynamic network
topologies. Bootstrapping in a MANET, in order to enable a node to join the network, is also a problem
since Gnutella assumes the use of bootstrap-servers or prior knowledge of other peers. To address
these shortcomings, the authors propose a cross-layer optimisation of the protocol — XL-Gnutella. In-
teraction between Gnutella and the routing agent is achieved by a cross-layer protocol stack design in
which a vertical interface extends the standard inter-layer primitives enabling both the application and
the network layer to react asynchronously to events generated by the other layer. XL-Gnutella extends
OLSR by allowing the routing agent to spread Gnutella peer credentials along with the routing control
packets, thus enabling efficient peer discovery and the selection of peers closer in the underlying phys-
ical network. The proposed cross-layer approach is able to tolerate the typical dynamic characteristics
of a MANET, providing a solution to the aforementioned shortcomings, while offering lower bootstrap
latency and higher query success rate. Although XL-Gnutella exploits cross-layer interaction, it focuses
on optimising the discovery and selection of neighbour peers and does not consider overlay routing or
search algorithms.
Another unstructured P2P system for MANETs which exploits cross-layer communication is proposed
in [38]. The MPP (Mobile Peer-to-peer Protocol) protocol suite consists of an application layer protocol,
MPP, an inter-layer communication protocol, Mobile Peer Control Protocol (MPCP), and a network rout-
25
ing protocol, EDSR (Enhanced Dynamic Source Routing). Given that MANETs already provide routing
capabilities, allowing the discovery of nodes by their IP address, MPP avoids implementing additional
P2P routing functionality at the application layer. Alternatively, EDSR extends DSR by adding new re-
quest and response types, allowing peers to be discovered based on other criteria besides their IP
address and therefore implementing P2P functionality at the network layer. This allows the P2P network
to automatically take into consideration changes in the physical network topology, concentrate routing
functionality within a single layer and reduce overhead and redundant information. At the application
layer, MPP handles the exchange of data between peers using the HTTP protocol. Additionally, MPP
also aims to provide the user with location aware services using the distance information, i.e. number of
hops, obtained via query replies. Handling the inter-layer communication between the application layer,
i.e. MPP, and the network layer, i.e. EDSR, is MPCP. MPCP provides a communication channel where
the application can register itself with EDSR, allowing it to initiate search requests and to receive, and
reply to, incoming search requests from remote peers. Although the authors do not provide simulation
results, some analytical considerations are presented, demonstrating the improved performance and
lower overhead that result from matching overlay topology with the underlying network topology through
inter-layer communication. Despite leveraging the synergies between the P2P application and MANET
routing algorithm, MPP does not contemplate important mechanisms, such as caching and replication,
used to limit the amount of traffic generated by query flooding. In addition, although EDSR is compatible
with DSR, legacy DSR nodes will drop the new request and response types rather than forwarding them,
which limits its interoperability with existing MANETs.
2.3.2 Structured Approaches
An implementation capable of efficiently supporting a DHT abstraction in a MANET is presented in
[39]. Ekta integrates a DHT, Pastry, with a MANET routing protocol, DSR, at the network layer, exploiting
interactions between both protocols. Integration of the physical and logical namespaces is achieved
through a one-to-one mapping between a node’s IP address and its nodeId which enables the use of
a single integrated routing structure to optimize routing performance. Ekta maintains a routing table
and a leaf set similar to Pastry where routing table entries are chosen based on their physical proximity
among other possible candidates. Entries are continuously updated, based on their proximity, using
overheard routes and active discovery methods. Unlike Pastry, each entry in these routing structures
contains the necessary source route to reach a given nodeId, rather than an IP address. Multiple source
routes per entry are maintained to increase robustness and efficiency. Pastry’s prefix-based routing is
used to deliver a message with a given key k to the node whose nodeId is numerically closest to k. If
no source route exists for the next hop along the overlay route, a route discovery process is initiated by
Ekta. When compared to a traditional layered approach using Pastry and DSR, Ekta provides a higher
26
packet delivery ratio with a lower average delay while incurring comparable overhead. Even with such
advantages, Ekta does not consider physical proximity in the construction of its DHT, it only optimizes
existing routes, replacing them with closer routes that it may overhear. This can still lead to long and
inefficient routes.
Another DHT substrate designed explicitly for MANETs is proposed in [40]. Like Ekta, MADPastry
integrates the functionality of a DHT and an ad-hoc routing protocol at the network layer to provide
efficient indirect routing in a MANET. It combines the Pastry DHT with the AODV routing protocol and
explicitly avoids the use of broadcasts where possible. Since conventional DHTs are unaware of the
underlying physical network and can produce long and inefficient physical paths, MADPastry, unlike
Ekta, considers physical locality in the construction of its DHT. This is achieved by using the concept of
random landmarking to create overlay clusters of physically close nodes. Nodes within a cluster share
a common overlay id prefix. A given node is temporarily considered a landmark node if it is currently
responsible for one of the landmark keys defined in the overlay. Other nodes associate themselves
with the landmark node that is closest to them, adopting its overlay id prefix. Nodes will periodically
check for closer landmark nodes, which, if found, cause a node to rejoin the overlay network with a
newly generated overlay id. Three different routing tables are maintained in MADPastry: a standard
AODV routing table for physical routes, a stripped down Pastry routing table and a standard Pastry leaf
set. The stripped down Pastry routing table is only required to maintain a reference to an arbitrary
node within each overlay cluster. Intra-cluster routing is then performed using the leaf set. This routing
approach sacrifices Pastry routing performance to drastically reduce maintenance overhead. The only
proactive routing table maintenance performed by MADPastry is the periodically checking with the left
and right leaves of the leaf set, which is required to guarantee overlay routing convergence. Results are
shown that demonstrate the significant performance advantage of MADPastry over a similar integrated
approach using Pastry and AODV without random landmarking.
27
3A Peer-to-Peer Middleware for Mobile
Ad-hoc Networks
Contents
3.1 Requirements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29
3.2 Network Architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30
3.3 Peer Architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35
28
The proposed P2P middleware is presented and detailed in this chapter. To begin with, an overview
of the main requirements for the middleware is provided, along with a set of basic assumptions about the
scenarios in which it will operate. A comprehensive description of both the network architecture and the
peer, or system, architecture follows, detailing the most important design decisions and the functionality
of the different middleware components.
3.1 Requirements
As discussed in Section 2.1.1, the dynamic and decentralised nature of MANETs, limits the type
of P2P overlays which can be constructed to support P2P applications. Thus, the first and foremost
requirement the middleware and the corresponding P2P overlay should satisfy is the ability to operate in
a decentralised manner, whereby each peer is of equal importance and no central peer or peers exist.
The middleware should be applicable to scenarios, which, due to higher node mobility, churn and/or
network partitions, are unsuitable for use with structured P2P overlays, characterised by a higher over-
lay maintenance overhead in the face of dynamic network conditions [21]. Furthermore, the failure or
departure of a peer from the overlay should have a negligible impact on the remaining peers and should
not require extensive network reconfiguration or a significant increase in signalling traffic.
Considering the fundamental importance of search functionality in P2P applications, the middleware
should provide such applications with the ability to search for a variety of resources available throughout
the network. In order to support a broad set of content distribution, collaboration and communication
applications, these searchable resources should not be limited to files, but also include services, peers
and records, thus providing greater flexibility than the file-centric P2P systems discussed in the previous
chapter. Such search functionality should include an appropriate query language so as to enable P2P
application users and/or developers to easily express the properties of the desired information. Given
the popularity of web search engines among users, the query language should provide a familiar syntax,
supporting keywords, partial match queries and boolean queries.
Although ideally all nodes of a given MANET would be fully aware of the P2P middleware, this
would be impractical in a real-world scenario and would significantly hinder interoperability among nodes
running different software stacks. As a result, the middleware should be able to operate correctly in a
heterogeneous MANET in which it is run on only a subset of the node population. The presence of the
middleware should be transparent to nodes which are not participating in the P2P network. Also, it is
assumed that a standard MANET routing protocol is required in order to support regular, non-middleware
enabled, TCP/IP applications.
A set of Application Programming Interfaces (API) should be provided by the middleware so as to
simplify the process of developing P2P applications. These APIs should not only expose the previously
29
mentioned search functionality, but also provide access to local content storage and indexing, application
registration and communication with other peers. Finally, a modular and extensible architecture should
be employed, where possible, in structuring the middleware to allow components to be easily replaced
or extended in order to improve functionality and the applicability of the middleware beyond the usage
scenarios initially envisioned.
A significant concern related to dynamic multi-hop wireless networks such as MANETs, is the perfor-
mance of large file transfers between nodes. Node mobility and link quality have a significant impact on
the throughput achieved by traditional stream-oriented reliable transport protocols such as TCP. Further-
more, performance and stability of a connection in a MANET degrade as the distance between nodes,
or the number of intermediate nodes, increases, leading to additional retransmissions. Therefore, the
middleware should avoid transferring files from distant peers if other peers, closer by, also provide the
same file. It must also deal with link breakage when transferring said files, as this is common occurrence
in MANETs, and resume file transfers accordingly.
3.2 Network Architecture
The proposed solution combines the use of an unstructured P2P overlay, because of its lower overlay
maintenance overhead in the presence of frequently changing network conditions and support for a
more comprehensive set of search queries when compared to a structured P2P overlay; with a reactive
MANET routing protocol, on account of its greater efficiency in terms of control overhead and increased
suitability for networks with moderate mobility when compared to other flat ad-hoc routing protocols.
HWMP was chosen as the MANET routing protocol due to its inclusion in the IEEE 802.11s draft,
a mesh networking amendment to the current IEEE 802.11 standard [30]. IEEE 802.11s promises
to become the de facto standard for the creation of multi-hop wireless mesh networks and MANETs.
Although HWMP is a hybrid routing protocol, incorporating both reactive and proactive components,
only its reactive component is considered. HWMP’s reactive component is based on a modified version
of the well known AODV [26] routing protocol, adapted for link-layer address-based routing and radio-
aware link metrics, with comparable performance characteristics. In HWMP, link-layer routes are also
referred to as paths to distinguish them from their network-layer counterparts. No distinction is made
throughout the remainder of this document, hence both terms are used interchangeably.
A P2P overlay and the underlying multi-hop MANET share two common objectives: the discovery of
other nodes in the network according to a specific criteria and the routing of packets between nodes.
The MANET routing protocol provides the ability to discover other nodes in the network based on the
destination node’s address. This functionality is integrated in the route discovery process. Similarly, P2P
overlays provide a mechanism to discover peers whose contents match a given query, such as a set of
30
QueryQuery Hit
Overlay linkRoute RequestRoute Reply
Figure 3.1: Searching in an unstructured P2P overlay (left) and route discovery using a reactive MANET
routing protocol (right)
keywords or a key, and route messages between peers.
By considering a typical unstructured P2P overlay and a MANET with a reactive routing protocol, the
similarities are even more pronounced as can be seen in Figure 3.1. In both cases, the discovery of
a peer or node is done on-demand. Controlled broadcasting is used to propagate the search or route
request throughout the network. Each peer or node broadcasts the request to its set of network neigh-
bours, which will in turn broadcast it to their neighbours until a given termination condition is reached.
As the request is propagated throughout the corresponding network, a network path is incrementally
constructed. Upon reaching a peer whose contents match the search query or a node whose address
matches the requested destination, a reply is unicast to the requester using the discovered reverse path.
Given the parallel between both types of networks, the proposed solution avoids duplicating the dis-
covery and routing functionality at different layers in the network stack by integrating the process of
searching for information in the P2P overlay with the discovery of the network routes in the underlying
MANET. Rather than use a P2P overlay network consisting of a set of persistent connections setup be-
tween possibly distant peers in order to route application layer search requests and replies, the MANET
routing mechanisms are used instead. This is achieved by extending the HWMP path request (PREQ)
and path reply (PREP) messages in order to transport, respectively, application layer search requests
and search replies. Therefore, one of the primary services provided by the middleware, which will be
described in further detail in Section 3.3, consists in allowing P2P applications to submit new search
requests to the network routing layer and notifying said applications of received search requests.
31
This integrated approach offers several advantages over the use of two distinct networks. The
MANET is exclusively responsible for the maintenance of the network topology so that any changes
introduced by network participants, such as nodes failing or moving in and out of range of other nodes,
are immediately reflected in the P2P overlay. The P2P overlay does not suffer from a high path stretch,
defined as the ratio of physical hops traversed on the path between two peers in the overlay to the num-
ber of physical hops on the direct unicast path between those peers, because overlay messages are
forwarded along the latter. Thus, no mismatch exists between the P2P overlay network topology and
the underlying MANET topology. Furthermore, traffic generated for overlay maintenance purposes is
reduced, as peers do not need to maintain persistent connections to other peers in the network. Finally,
the packet overhead for application layer searching and the subsequent route discovery is lower than if
both tasks were performed separately.
To illustrate the operation of the proposed solution, consider the multi-hop MANET shown in Fig-
ure 3.2 .Let P = {A,B, C, D, E} be the set of peers in the network and S = {1, 2, 3, 4} the set of
searchable documents stored in the network. Multiple copies of the same document may be available
from different peers. Furthermore, consider an application at peer A which issues a query matching
documents 3 and 4. The application starts by informing the middleware of the search query. An HWMP
PREQ frame containing a search request is generated by the middleware and broadcast using HWMP
to the set of physical neighbours, i.e. the peers which are within radio transmission range: B and C.
Each neighbour peer, upon receiving the PREQ frame, informs the middleware of the received search
request before any further forwarding action is taken. Also, HWMP stores peer A’s address as the next
hop on the reverse path to the requesting peer. The middleware searches its local document repository
for any documents matching the query contained in the search request. Since neither peer has a match-
ing document, the middleware informs HWMP of its forwarding decision and the PREQ frame is again
broadcast to all neighbouring peers. Propagation of the search request throughout the network is limited
by the time-to-live (TTL) value defined for the enclosing PREQ frame and duplicate search requests are
discarded at each peer. Finally, the PREQ containing the search request reaches peer D. HWMP stores
peer B’s address as the next hop on the reverse path to the requesting peer, peer A. Since matching
documents are available at peer D, the discovered reverse path is used to unicast a HWMP PREP frame
containing a search reply back to peer A. This search reply includes information necessary to contact
peer D, such as its network address and transport layer ports, along with the actual search results. It is
important to note that the size of the search results which can be included in this search reply is limited
by the maximum transmission unit (MTU) of the underlying IEEE 802.11 network interface, thus, the re-
questing peer may need to use an alternate mechanism to obtain the complete result set. As the PREP
is unicast to peer B and subsequently to peer A, the forward path between both peers is discovered.
The behaviour of peer E is analogous as it too will generate a PREP containing a search reply which will
32
B
HWMP PREQ with search request
A
D
E
HWMP PREP with search reply
C
Documents: 3, 4Documents: 2
Documents: 4Documents: 1, 2
Figure 3.2: Searching for documents in a MANET by leveraging HWMP route discovery
then be unicast back to peer A through peer C.
Peer A, upon receiving both PREPs containing the search replies, knows of one source peer, peer
D, for document 3 and two source peers, peer D and peer E, for document 4. Additionally, as path
discovery was executed in parallel to the searching process, valid bidirectional network paths are now
available for reaching peer D and peer E from peer A and vice-versa. The application can use the
information contained in the search replies to contact the peers and obtain the requested documents.
Further communication between peers is achieved by using the discovered unicast paths along with
an appropriate transport protocol. If subsequent changes to the network topology cause said paths to
break, HWMP’s path failure notification and discovery mechanisms are used to discover alternate paths.
In the case of document 4, which is available from multiple peers, the application is faced with an
additional problem: deciding which peer it should contact to transfer the document from. Without any
further knowledge of the underlying network, the application would choose randomly between peers D
and E. As discussed in Section 3.1, this is not a desirable solution since the number of hops between
peers and the aggregate link quality over all hops on a given network path affect the achievable transfer
throughput. In general, peers which are physically closer and can be reached with fewer hops are
preferable.
A naive optimisation consists in using the time elapsed between transmitting the search request and
receiving the corresponding search reply from each peer, given that the search reply sent by the peer
closest to the requester would usually be received first. Unfortunately, this value is highly sensitive to
transient queuing delays at intermediate nodes and to the time required to match a query and generate
33
B
A
D
EC
8
3
105
Peer Path metric
15
D
E
11
Figure 3.3: Source peer selection using HWMP path metrics
a reply. The latter depends on a peer’s processing capacity and current load. An alternative optimisation
relies on the hop count, i.e. the number of hops on the path between two nodes. The peer whose path
has the lowest hop count would therefore be selected as the source for the document transfer. Although
unaffected by the problems mentioned for the previous optimisation, the hop count can still be a mis-
leading selection criteria as it does not take into account the characteristics of the links between nodes.
Moreover, in the aforementioned example, the information provided by the hop count is insufficient for
selecting a peer as the paths from peer A to peers D and E have the same hop count, 2.
HWMP uses a link metric when building and selecting network paths [41]. Upon reaching their
destination, PREQ and PREP frames contain the cumulative value of the link metric calculated at each
hop along traversed path. This value is stored along with the remaining forwarding information for each
path. IEEE 802.11s defines a default link metric to ensure compatibility, the airtime link metric, which
defines the cost, i.e. the amount of network resources consumed, to transmit a frame over a given
link [42]. The proposed solution uses the path metric for discovered paths, stored in HWMP’s forwarding
information table, as a criteria for selecting a source peer for a given document.
Consider the network paths and corresponding link metrics shown in Figure 3.3, discovered as a
result of the search process described in the aforementioned example. Furthermore, consider the use
of a bi-directional airtime link metric, that is, the cost of transmitting a frame from peer X to peer Y is
equal to the cost of transmitting the same frame from peer Y to peer X. Peer A, given the availability of
two source peers for document 4, would consult HWMP’s forwarding information table to obtain the path
metric for both destinations: 8 + 3 = 11 in the case of peer D and 10 + 5 = 15 in the case of peer E.
Due to the lower cost of the path from peer A to peer D, the application would opt to transfer document 4
from peer D. The middleware thus provides P2P applications with an optimised, network-aware, source
peer selection service when multiple sources are available for the same content.
34
In the previous example, the set of nodes that form the MANET coincides with the set of peers in the
P2P overlay. As mentioned in Section 3.1, this is rarely the case in real world scenarios. As a result, two
types of nodes can be distinguished for a MANET on top of which the middleware is deployed: the nodes
which execute the middleware, henceforth known as P2P nodes, and the nodes which do not execute
the middleware, henceforth known as legacy nodes. For both types of nodes, HWMP is employed as the
MANET routing protocol. Due to the use of HWMP routing mechanisms to propagate search requests
and replies, the legacy nodes must satisfy only one condition for correct P2P network operation: any
application layer payload contained in received PREQ or PREP frames must be included if the frame
is forwarded. This provides a transparent fallback mechanism, ensuring that legacy nodes forward the
search request or reply toward their destination despite not running the middleware. Satisfying this
condition is dependent on the selected HWMP implementation, hence, this issue will be discussed in
further detail in Chapter 4.
Finally, since a single MANET can have distinct groups of users with different requirements, the set
of P2P nodes can be further divided into subsets known as P2P groups. P2P nodes belonging to the
same group share a common group identifier. The use of P2P groups allows peers to limit the extent
of their search requests, i.e., only peers from the same group will reply to a given request, as well as
establishing private groups of peers with secure searching and communication capabilities.
3.3 Peer Architecture
The P2P middleware provides the functionality required to establish the network architecture de-
scribed in the previous section. An overview of the peer architecture is presented in Figure 3.4, where
the middleware is introduced between the application and the existing operating system services. The
middleware is further divided into a set of interconnected components, each with a well defined set of
responsibilities:
• P2P-HWMP (Section 3.3.2) - Provides an interface for use by the remaining middleware compo-
nents, namely the query routing service, in order to send, and be notified of, search requests and
search replies contained in HWMP messages.
• Storage service (Section 3.3.3) - Provides a local repository for storing and indexing application
content in addition to being responsible for resolving queries, such as those received in search
requests, to determine whether any matching content is available.
• Query routing service (Section 3.3.4) - Responsible for generating new search requests con-
taining application supplied search queries, deciding on how received search requests should be
35
Application
Middleware communication service
Peer Management
Service
Peer Registration
Service
Peer Selection Service
Storage Service
Query Routing Service
Operating System
P2P-HWMP
HWMP
Figure 3.4: Peer architecture
forwarded by the underlying routing protocol, HWMP, and generating search replies in response to
received search requests.
• Service registration service (Section 3.3.5) - Manages the registration of services provided by
P2P applications and the middleware, allowing them to be discovered by other peers in the net-
work.
• Peer selection service (Section 3.3.6) - Performs source peer selection using HWMP path metrics
and maintains a list of known peers in the network.
• Peer management service (Section 3.3.7) - Provides configuration information for other middle-
ware services and enables remote monitoring of the middleware.
• Middleware communication service (Section 3.3.8) - A P2P application interacts with the middle-
ware through an API provided by the middleware communication service. This service is respon-
sible for dispatching application requests to the appropriate middleware service and for enabling
generic middleware to middleware communication, beyond that which is provided by the query
routing service.
Other than these components, HWMP must be extended to support middleware supplied information
and interaction with the middleware.
To exemplify the interactions between the various components, consider a scenario similar to the
one presented in Section 3.2, where a P2P application wishes to search the network for documents
matching a given query string. The application starts by calling the search function provided by the
middleware API and passing an appropriate query string, as illustrated in Figure 3.5. The middleware
36
Application
Middleware communication service
P2P-HWMP
HWMP
Query routing service
QuerySearchRequest
Query
QuerySearchRequest
PathRequest
MANET
search("the quick brown fox")
Figure 3.5: Interactions between middleware components at the requesting peer
communication service dispatches this request to the corresponding service, the query routing service,
which will generate a new search request containing the query string. The newly generated search
request is sent to P2P-HWMP along with the indication of the TTL to use for the enclosing PREQ frame
and the address to which this frame should be sent, typically the broadcast address. P2P-HWMP then
uses HWMP to construct and send a PREQ frame containing the search request.
A neighbour peer, upon receiving a PREQ frame, will contact P2P-HWMP for further processing,
as illustrated in Figure 3.6. P2P-HWMP extracts the search request from the PREQ frame and notifies
the query routing service, which in turn contacts the storage service in order to determine if any locally
stored documents match the query string contained in the search request. If no documents match the
received query, the query routing service notifies P2P-HWMP of its decision to continue forwarding the
search request, along with the address to which the search request should be forwarded. Otherwise,
if any documents match the received query, the query routing service generates a new search reply
containing the peer’s contact information and, depending on their size, the actual search results. The
search reply is sent to P2P-HWMP which will in turn generate and send a PREP frame using HWMP,
containing the search reply, in response to the received PREQ. This PREP frame is unicast back to the
37
P2P-HWMP
HWMP
Query routing service
QuerySearchRequest
PathRequest
MANET
StorageService
QuerySearchRequest
Query
Results
ResultsSearchReply
ResultsSearchReply
PathReply
Figure 3.6: Interactions between middleware components at the receiving peer
requesting peer. Finally, once the PREP frame is received at the requesting peer, the enclosed search
reply travels up to the middleware communication service which ultimately returns the search results to
the P2P application.
A detailed description of each middleware component presented in Figure 3.4, as well as the required
modifications to HWMP, is provided in the following sections.
3.3.1 Extending HWMP
The inclusion of middleware supplied information alongside HWMP control messages, namely path
requests and path replies, requires the extension of HWMP and the frames used to transport its control
messages.
HWMP control messages are transported in IEEE 802.11 management frames of subtype action
[43]. These action frames, which follow the standard format defined for IEEE 802.11 MAC frames,
contain a frame body consisting of an action field, describing the management action, and, optionally,
one or more vendor-specific information elements. An information element refers to the encoding of
a variable length field, in type-length-value format, which is used to store management information.
38
HWMP control messages are stored in information elements. Furthermore, a vendor-specific information
element allows non-standard management information to be included in management frames. Such
information elements are distinguished by an added field containing the IEEE Organisationally Unique
Identifier (OUI) of the vendor or organization responsible for defining the element’s contents.
Given the aforementioned action frame format, two methods are available for including additional
information alongside HWMP control messages: the extra payload can be included in the same infor-
mation element as used for the control message or it can be included in an additional vendor-specific
information element. It is important to note that according to the IEEE 802.11 standard, the information
stored in an information element is limited to 255 bytes. In case of the former method, this severely
restricts the space available for the extra payload, which is already partially filled by the control mes-
sage. Thus, the latter method is employed, whereby one or more vendor-specific information elements
are used to transport the additional information. The number of vendor-specific information elements
which can be included in a given action frame is limited only by the maximum frame size supported by
the underlying network.
In this particular case, the middleware only requires the ability to transport search requests alongside
PREQ messages and search replies alongside PREP messages. Consider the former, if the length of
the search request does not exceed the maximum length supported by a vendor-specific information el-
ement, as mentioned above, a single information element is constructed. Otherwise, the search request
is segmented into multiple information elements. The constructed information elements are inserted into
the corresponding action frame following the PREQ information element as shown in Figure 3.7. The
inclusion of a search reply in a frame containing a PREP message is analogous.
In addition to the format of the management frames used to transport control messages, it is impor-
tant to describe how entities external to HWMP, such as the middleware, interact with HWMP to spawn
and handle such frames. In general, the creation and subsequent transmission of PREQ and PREP
MAC Header PREQAction
(PREQ)
Element ID(Vendor IE)
OUI(XX:XX:XX)
Payload#1
Vendor IE #1
Vendor IE #n FCS
Length
...
24 1 39 257 4
1 1 3 252
6-257Octets:
Figure 3.7: Format of an IEEE 802.11 management frame used to transport an HWMP PREQ control
message and a middleware supplied search request
39
PREQ Handler PREP HandlerVendor OUI
CC:CC:CC
p2p_hwmp_parse_preq()
vendor_a_handler()
p2p_hwmp_parse_prep()
—
99:99:99
Figure 3.8: An HWMP vendor registry with two registered external entities
messages is triggered exclusively by events in the operating system’s network stack, for example, the
detection of a missing route for a certain destination or the reception of a given message. In order to
allow external entities to trigger the creation and transmission of these control messages, the corre-
sponding programming interfaces, used internally by HWMP, are exported. Using such interfaces, an
external entity can define the value of the various parameters, such as the destination or TTL, of the
PREQ and PREP messages it wishes to send and finally trigger the transmission of the corresponding
frame. Furthermore, the functionality of such interfaces is extended, allowing external entities to spec-
ify a set of vendor-specific information elements to be included in the transmitted frame alongside the
control message. These information elements will carry the external entity’s payload.
Similarly, external entities must be notified by HWMP of received PREQ and PREP frames containing
relevant vendor-specific information elements. To achieve this, a vendor registry is added to HWMP
such that each external entity, identified by its vendor’s OUI, registers itself with HWMP, specifying a
handler for each type of control message it is interested in processing. The use of an OUI to identify
external entities provides a convenient and compact solution since it can also be used to identify the
corresponding vendor-specific information elements without any additional fields. As a result, only one
external entity per vendor or organization can be registered with HWMP at a given time.
Figure 3.8 illustrates the structure of a vendor registry with two registered external entities. When
a control message is received by HWMP, the enclosing frame is scanned for any vendor-specific infor-
mation elements. For each information element, if a vendor with a matching OUI exists in the vendor
registry and a handler for that type of control message has been specified for that vendor, then the cor-
responding handler is called, informing the external entity of the contents of the information element and
the enclosing frame.
Handlers can be passive or active. On the one hand, a passive handler simply processes the con-
tents of the information element and returns control to HWMP for further processing and forwarding of
the enclosing frame. Thus, a passive handler does not interfere with the normal operation of HWMP.
Active handlers, on the other hand, assume responsibility for forwarding the enclosing frame to the next
hop or hops and may, if required, modify its contents. Active handlers are useful in cases where the ex-
ternal entity is better suited to decide how and if a frame should be forwarded. As an example, consider
40
a PREQ frame containing a search request which is broadcast to all neighbouring nodes at each hop.
The middleware could decide to forward this frame based on whether or not locally matching results
are available in order to limit the propagation of the search query. On top of that, active handlers allow
the implementation of more advanced query routing mechanisms on top of HWMP in addition to those
based on simple query broadcasting. This characteristic is discussed in further detail in Section 3.3.4.
As a result of aforementioned extensions to HWMP, the middleware at a given node can trigger, as
required, the transmission of PREQ and PREP frames containing, respectively, search requests and
search replies. The middleware will also register a handler for PREQ and PREP frames with the HWMP
vendor registry in order to be notified of any frames containing search requests or replies. Finally, in the
case of PREQ frames, the use of active handlers allows the middleware to carry out its own forwarding
decisions.
It is important to note that these extensions to HWMP are sufficiently generic so as to enable other
uses aside from the P2P middleware solution described in this work. In fact, any application which
benefits from disseminating information as part of the route discovery process can take advantage of
the described functionality.
3.3.2 P2P-HWMP
P2P-HWMP is responsible for sending search requests and replies, received from the query rout-
ing service, to the appropriate peers using HWMP. It is also responsible for notifying the query routing
service of any received requests and replies, and, in the case of search requests, acting on the for-
warding decisions provided by the query routing service. P2P-HWMP uses the mechanisms described
in Section 3.3.1 to interact with HWMP. In spite of this, it abstracts the details of the underlying HWMP
implementation in the functionality it provides to the query routing service.
In order to send a new search request, the query routing service contacts P2P-HWMP with the
request along with a maximum TTL, i.e., the number of hops after which the request should be dropped
by HWMP, and the address or addresses of the next hop to which the request should be sent, such
as, the broadcast address. P2P-HWMP segments the search request by creating a set of one or more
IEEE 802.11 vendor specific information elements according to the method described in Section 3.3.1.
These information elements share P2P-HWMP’s OUI, 99:99:99, which was chosen arbitrarily among
the set of unregistered IEEE OUIs. HWMP is used to create a new PREQ frame, with the specified TTL,
containing the constructed information elements. Unlike a standard path request, which specifies the
address of the destination node to which a path should be discovered, this request uses a destination
address guaranteed to be different from the address of any other node in the network. This ensures that
no node can consider itself to be the final destination of the path request as this would stop the request
from being propagated any further. Finally, the created PREQ frame is sent using HWMP to the address
41
or addresses specified by the query routing service.
HWMP P2P-HWMP Query RoutingService
p2p_hwmp_handle(PREQ)
notify_request(request)
decision(forward, ff:ff:ff:ff:ff:ff)
send(PREQ, ff:ff:ff:ff:ff:ff)
send_reply(reply)
send(PREP, 08:11:14:ef:fa:ff)
Figure 3.9: Interactions performed by P2P-HWMP for a received PREQ frame
P2P-HWMP registers two handlers with the HWMP vendor registry, one for each frame type. These
handlers are called to notify P2P-HWMP of received PREQ and PREP frames containing its information
elements. The PREQ handler is defined as an active handler and the PREP handler as a passive han-
dler. When a PREQ frame, containing information elements matching P2P-HWMP’s OUI, is received by
HWMP, the PREQ handler is called. An example of the interactions performed by P2P-HWMP’s PREQ
handler when processing a received PREQ frame is presented in Figure 3.9. This handler temporarily
stores the PREQ frame and notifies the query routing service of the enclosed search request. No for-
warding action is taken for the PREQ frame until a forwarding decision is received from the query routing
service. The frame can either be dropped or forwarded. To forward the stored frame, P2P-HWMP incre-
ments the hop count, decrements the TTL and sends the frame using HWMP to the next hop address
or addresses specified in the forwarding decision. Due to the use of an active handler for PREQ frames,
P2P-HWMP assumes the responsibility for forwarding such frames if required.
If matching content exists at the current peer, the query routing service will also contact P2P-HWMP
42
with a search reply to be unicast back to the requesting peer. As with the search request, the search reply
is segmented into a set of vendor specific information elements. The previously stored PREQ frame,
namely the address of the requesting peer, is used to construct a new PREP frame containing these
information elements. The PREP frame is sent by HWMP along the discovered path to the requesting
peer.
Lastly, the PREP handler is called for received PREP frames whose vendor specific information ele-
ments match P2P-HWMP’s OUI. This handler merely notifies the query routing service of the enclosed
search reply. If the current peer is not the final destination of the PREP frame, it is forwarded by HWMP
to the next hop on the path to its destination. Since a passive handler is used for PREP frames, P2P-
HWMP does not interfere with the forwarding of PREP frames by HWMP.
3.3.3 Storage Service
P2P applications use the storage service to index, store and search their content. In order to support
a wide range of applications, as mentioned in Section 3.1, the concept of document was chosen as
the primary unit of storage. A document is an object consisting of a set of named fields whereby each
field has an associated value. Documents can have varying fields of different types including strings,
numbers, booleans and dates, among others. The use of documents as a primary unit of storage places
as few restrictions as possible on the type and structure of application content.
To exemplify the use of documents in representing application content, consider a P2P social net-
working application which allows users to create and manage their profile. Users can then search the
network for other users whose profile contains, for example, similar interests. A possible document rep-
resentation of a user’s profile is presented in Figure 3.10. Note that each profile document can have
a different set of named fields. A user might not want to disclose his or her age so the corresponding
field would be omitted. Similarly new fields which weren’t initially envisioned when the application was
developed can easily be added to new and existing documents.
The storage service thus presents P2P applications, as well as the remaining middleware compo-
nents, with a document-oriented database interface for storing and retrieving documents. The provided
"name" : "John Smith"
"age" : 30
"country" : "US"
"state" : "CA"
"interests" : ["Ski", "Fishing", "Cooking"]
Figure 3.10: Example of a document used by a P2P social networking application
43
"name:John AND NOT country:UK"
"Ski John CA"
"interests:Ski AND age:30"
Figure 3.11: Example of the types of search queries supported by the storage service
functionality includes adding new documents, updating and removing existing documents and searching
for documents matching a given search query. The latter relies on the use of an indexing mechanism
when documents are added or modified in order to allow efficient retrieval of matching documents. Each
application can request the creation of a database for storing its own documents and, optionally, specify
a password which is used by the storage service to authenticate future database modification requests.
The created databases are schema-free, i.e., documents of the same type may have a different struc-
ture, and have a flat address space, with each document being identified by a unique identifier.
The interaction between an application and the storage service is carried out through the middleware
API. A typical interaction starts with the creation of a new document database, followed by the addition
of a set of documents, with each newly added document being assigned a unique identifier. This unique
identifier can be used to retrieve, update or remove a specific document from the database. When a
document is added to the database, the indexes for a given subset of its fields are updated, ensuring
fast query resolution. In order to allow other instances of the application, running on remote peers, to
search for documents contained in the database, an application must explicitly mark its database as
being shared. In that case, the query routing service, upon receiving a search query for the specified
application from a remote peer, will contact the storage service in order to determine if matching doc-
uments are available at the current peer and, if so, return the matching results to the requesting peer.
Applications can, at any time, toggle whether or not to share their document databases.
The query syntax supported by the storage service is similar to that which is employed in mainstream
web based search engines. This allows P2P application developers to quickly leverage the search
functionality provided by the document database without having to learn an entirely new syntax. That
being so, a query is composed of a set search terms connected by the boolean operators AND, AND
NOT, OR and XOR. If no boolean operator is specified between two consecutive search terms, the OR
operator is assumed. Optionally, a search term or a set of search terms may be prefixed with a field
specifier, restricting it to a specified field. Recall the document in Figure 3.10 which represents a user’s
profile in a social networking application. Figure 3.11 shows three search queries which could be used
to match this document. The first query contains two field specifiers restricting the search terms ‘John’
and ‘UK’ to the fields name and country respectively. This query would match any users named John
which are not from the United Kingdom. The second query contains three search terms and matches,
since no boolean operators are specified, any documents containing the terms ‘Ski’, ‘John’ or ‘CA’ in
44
"dirname": "/tmp"
"extension": ".iso"
"filename": "debian-502-i386-DVD-1.iso"
"mimetype": "application/octet-stream"
"modification_time": 1246202385
"size": 4697108480
"tags" : "debian linux lenny iso"
Figure 3.12: Example of a file-backed document
any of its indexed fields. The use of this query syntax need not be limited to application developers
and can be exposed to users of such applications, where applicable, to allow users to directly query the
application’s database.
Although the document-oriented storage approach is applicable to a wide range of applications, file
sharing applications represent an important distinction. In such applications, files already exist in the file-
system and storing an additional copy in the document database would not only be inadequate but would
also degrade database performance. As such, a special document type, the file-backed document, is
introduced to describe a file residing in the local file system. A file-backed document consists of a set
of mandatory fields such as the file’s name, location in the file-system, type and size. An example of a
file-backed document is presented in Figure 3.12. Consequently, a file-sharing application will start by
creating a file-backed document for each file it wishes to share. The resulting file-backed documents can
have any number of fields appended, such as the tags field in Figure 3.12, before finally being added to
the application’s document database. Note that file-backed documents are handled identically to other
types of documents throughout the middleware, with the exception of the middleware communication
service which will be discussed in further detail in Section 3.3.8.
3.3.4 Query Routing Service
When an application wishes to search the network for documents matching a given query it interacts
with the query routing service through the middleware API. The query routing service is responsible
for disseminating the search query throughout the MANET, obtaining results from peers which contain
matching documents and processing these results so they can be returned to the application. Further-
more, at each intermediate peer, the query routing service is also responsible for deciding if and how a
received search request should be forwarded to neighbouring nodes, thereby allowing the middleware
to affect the routing of the corresponding HWMP PREQ frame.
An application supplied search query consists of a query string which conforms to the syntax de-
45
scribed in Section 3.3.3. In order to send this query string to another peer, a search request message
is constructed which contains, in addition to the query string itself, information required to contact the
requesting peer, namely its network address and the port used for the middleware communication ser-
vice, as well as a unique message identifier used to detect duplicate messages at receiving peers. This
search request is sent to P2P-HWMP by the query routing service for inclusion in a new HWMP PREQ
frame which will be propagated throughout the network. The length of the query string included in the
search request is limited by the MTU of the underlying IEEE 802.11 network interface used to transmit
the HWMP PREQ frame.
Upon receiving a search request via P2P-HWMP, the query routing service at the receiving peer
must first decide on whether or not it should continue forwarding the search request. For example, if the
receiving peer contains documents matching the enclosed query, it may choose to stop propagating the
query to avoid further network resource consumption. If the search request is to be forwarded to other
neighbouring nodes, be they P2P nodes or legacy nodes, as mentioned in Section 3.3, the query routing
service must also decide to which of these neighbouring nodes the request should be forwarded. This
decision also applies to the requesting peer for the initial transmission of the search request. Both these
decisions, whether to forward the received search request and to which nodes such request should
be forwarded, are collectively referred to as the forwarding decision and constitute the query routing
protocol. Various query routing protocols can be supported by the query routing service without changes
to the underlying MANET routing protocol. Once a decision has been taken, the query routing service
must communicate such decision to P2P-HWMP in order to trigger the appropriate forwarding action.
The query routing service enables a set of query routing protocols similar to the search strategies
described for unstructured P2P overlay networks in Section 2.1.2. Examples of possible query routing
protocols include query flooding, expanding ring search and random walks. Given the lack of correlation
between a peer and the documents it stores, these protocols aim to maximise the probability of locating
matching documents by covering an increasing number of peers.
In the case of query flooding, the behaviour matches that of conventional unstructured P2P systems
and reactive routing protocols, i.e. a search request is broadcast, at each node, to all neighbouring
nodes until a maximum number of hops is reached which is specified by the request’s TTL. The TTL
is used to limit the propagation of the search request and avoid flooding the entire network. Although
providing the fastest way to cover a large number of nodes and yielding the shortest path, this method
strains the available network resources and suffers from the need to select a fixed TTL which can provide
an acceptable trade-off between network coverage and resource consumption.
Expanding ring search consists of an optimisation of the query flooding strategy. A search request
is broadcast, at each node, to all neighbouring nodes but, rather than relying on a fixed TTL, the search
process is split into a set of iterations with increasing TTL values. The first iteration propagates the
46
search request using a low value for the TTL in order to search only within a few hops of the requesting
node. If no reply is received, the TTL is incremented and a new iteration is started with the retransmission
of the search request by the requesting peer. Each iteration covers an increasing number of nodes and
increases the probability of finding the requested documents. This method provides a clear advantage
for popular content, which due to its higher replication rate, can usually be found near the requesting
peer. Expanding ring search limits the query flooding necessary to find such content at the cost of
increasing the query delay and, in the case of rare content, the consumption of network resources.
Finally, a random walk consists in randomly choosing, at each node, a single node from the set of
neighbouring nodes as the next hop for the received search request, excluding the node this request was
received from. A random walk does not rely on the use of broadcast traffic to propagate a search request
throughout the network thereby preserving network resources. Since node coverage only increases
by a single node each time the query is forwarded, query delay is higher than both of the previously
mentioned approaches and longer paths are followed to reach the replying peers. Selecting a query
routing protocol depends on many factors, namely scalability, the trade-off between the query delay
and resource consumption and the popularity of the desired content. To simplify the description of the
remaining functionality of the query routing service, consider the use of query flooding as the query
routing protocol.
In addition to providing P2P-HWMP with forwarding decisions for received search requests, the query
routing service contacts the storage service to determine if the current peer contains documents match-
ing the received query and, if so, generates a search reply which is sent back to the requesting peer.
The search reply adopts a similar structure to the search request by including the information required
to contact the replying peer, necessary in order to transfer documents or further results. The reply also
includes a unique message identifier which is set to the same value as the message identifier of search
request it is replying to. This allows the requesting peer to match replies to the corresponding request.
Lastly, the search reply contains a result set consisting of a list of documents matching the query. In-
stead of including the entire document, as stored by the storage service, in the result set, the query
routing service obtains an abridged copy of the document containing only the set of fields considered
relevant for displaying or processing the results by requesting application. Once constructed, the search
reply is sent to P2P-HWMP which will in turn create the enclosing HWMP PREP frame and send it to
the requesting peer.
As an example, consider a P2P application which allows users to discover music tracks shared by
other users on the network. The developer of this application could choose to include only the artist,
album and track name fields for documents representing tracks contained in the result set, and exclude
other fields, such as the track’s lyrics, which are not relevant when displaying the list of matching tracks to
the user. If the user wishes to obtain further information regarding a selected track, the application would
47
then contact the replying peer to transfer the remainder of the document and all of its fields. By returning
only the relevant fields, significant bandwidth savings can be achieved when searching for documents
which exhibit a large number of fields or fields which are of reduced interest to the requesting application.
Developers can easily define, through the middleware API, the set of fields used to generate copies of
documents stored in an application’s document repository and returned as part of result sets for that
application. If the requesting peer receives an incomplete copy of a document as part of the result set,
it can use the document’s unique identifier, included in any copy of the document, to transfer a complete
copy from the remote peer using the middleware communication service described in Section 3.3.8.
Ideally, all possible result sets would be sent using the search reply but, as is the case with the query
string, the size of the result set is limited by the MTU of the underlying IEEE 802.11 network interface
seeing that the enclosing search reply is transported in an HWMP PREP frame. Rather than impose
a limit on size of the result set, which would severely constrain the number of matching documents
returned, an out-of-band mechanism is employed to obtain the result set in cases which prevent its
inclusion as part of the search reply. Various potential out-of-band mechanisms exist which could be
used to obtain the result set. The query routing service could split the result set into one or more UDP
packets and send these to the requesting peer in parallel with the search reply. Although the requesting
peer would not have to explicitly request the result sets from the replying peers, and thereby suffer an
additional delay, it would be flooded by result set packets if each of these peers were to reply with a
large result set. UDP’s lack of congestion control only exacerbates this problem. To avoid flooding
the requesting peer by implicitly sending the result set, an alternative approach could be employed in
which the requesting peer would explicitly request each segment of the result set from a replying peer.
A description of the number and size of these segments would be included in the search reply, thus
allowing the requesting peer to control the number of inbound segments and stop transferring the result
set once enough matching documents have been received. Despite this, the aforementioned approach
still suffers from the limitations imposed by UDP: the need to reorder segments and reassemble the
result set, lack of congestion control and the loss of segments due the unreliable nature of the transport
protocol.
Since the middleware communication service already provides a reliable request/response protocol,
it is chosen as the mechanism used for obtaining result sets from replying peers. The query routing
service starts by constructing the result set for a given query, if this result set can not be included in
the search reply due to its size, it is temporarily cached and a unique result set identifier is generated
identifying the result set. This identifier is included in the search reply, alongside the number of matching
documents, in place of the actual result set. Upon receiving the search reply, the requesting peer
uses the middleware communication service to obtain the cached result set given the unique identifier
contained in the search reply. Result sets of any size can be reliably transferred using this mechanism.
48
Once enough unique documents have been received, the query routing service at the requesting
peer refrains from obtaining further result sets. If more than one source peer is available for a given
document the list of source peers is sorted using the peer selection service described in Section 3.3.6.
Finally, the matching documents are returned to the application along with the information regarding
their source peers.
3.3.5 Service Registration Service
The P2P middleware is not limited to providing an application with the means to search for documents
shared by other instances of the application on the network. In fact, many applications require the
ability to search for other applications and the services they provide. This service discovery capability is
enabled through the use of the service registration service which allows applications to register services
they provide with the middleware. Once registered, these services can be discovered by other peers in
the network.
Applications interact with the service registration service through the middleware API. The service
registration service maintains a document database destined exclusively for storing service description
documents. An application wishing to enable the discovery of a service would start by creating a service
description document, a special document type with two mandatory fields: the url, indicating the location
of the service on the network per RFC 3986 [44], and the type, used to identify services which share a
common protocol. It is important to note that, although a document can have any number of additional
fields, as mentioned in Section 3.3.3, allowing further service information to be provided, the specifica-
tion of a service description language for describing services is beyond the scope of this document. The
mandatory fields of a service description document, url and type, provide only the essential information
necessary to identify and contact another service on the network.
Once created, the service description document is used to register the service along with a lifetime
during which such registration should remain valid. The document is added to the service registration
service’s document database. Other peers can now search for the service description document, using
the same mechanisms mentioned in Section 3.3.4 and Section 3.3.3, in order to discover the service.
The service registration service periodically checks its database for expired registrations and removes
the corresponding service description documents, preventing other peers from discovering services
whose registration has expired. Alternatively an application can use a unique identifier, received upon
registering the service, to unregister the service at any time.
Discoverable services are not limited to those registered by P2P applications. Other middleware
services, namely the middleware communication service and peer management service, which are re-
motely accessible, are automatically registered with the service registration service and can be easily
discovered by other peers on the network. As a result, a peer can discover other instances of the mid-
49
dleware running on the network and, in the case of the peer management service, determine which of
those instances can be remotely monitored. The remotely accessible functionality provided by these two
services is discussed in Section 3.3.8 and Section 3.3.7 respectively.
3.3.6 Peer Selection Service
Source peer selection, as mentioned in Section 3.2, is achieved through the use of the peer selection
service which is also responsible for maintaining updated information about other peers in the network.
When multiple source peers are available for a given document, the middleware uses information
about the path to those peers, provided by HWMP, in order to determine the best peer to use for obtaining
the document. The best peer among a set of peers is defined as the peer to which the current peer has
the lowest value of the path metric, assuming the path metric represents a cost function. This value is
stored in HWMP’s forwarding information table for each destination to which a valid path exists. The
value of the path metric is calculated during HWMP path discovery by adding up the individual values
of the link metric calculated at each hop on the path between both peers. In this case, the airtime link
metric, which represents the amount of channel resources consumed by transmitting a frame over a
given link, is chosen due to its radio-aware nature and its use as the default link metric for IEEE 802.11s.
It is important to note that any link metric representing the cost of a given link, for example, energy
consumed, could be used instead. The airtime cost [42], ca, of a link is defined by
ca =(
Oca + Op +Bt
r
)1
1− ept
where ept is the bit error rate for a test frame of size Bt, transmitted at a modulation rate of r using
an IEEE 802.11 modulation type characterised by a given channel access overhead, Oca, and protocol
overhead, Op. If two paths have the same path cost, the one with the lowest hop count is preferred as it
usually introduces the lowest queuing and forwarding delay.
By using the path cost, calculated by HWMP for each destination, the peer selection service provides
the ability to sort a list of peers according to their corresponding path costs. This list, sorted in order of
ascending path cost, can be used by applications wishing to obtain a document from a set of available
source peers. The application would typically start by trying to contact the peer with the lowest path cost.
If it is unable to contact this peer, the next peer in the list would be selected and so forth. Whenever the
connection to the currently selected peer is interrupted, the application can either retry this peer or opt
for the next peer. This decision depends on the trade-off between time spent waiting for the currently
selected peer to respond versus using another peer with a higher path cost and, therefore, a greater
network resource usage and in most cases, lower connection throughput. As a result of this sorted peer
list, applications can optimise the transfer of documents for which multiple sources exist by opting for
peers which minimise the amount of channel resources used. This is particularly important in a MANET
50
where such resources are scarce.
Finally, the peer selection service also caches information regarding other peers in the network. For
each known peer, this information includes the peer’s unique identifier, network address and transport
protocol port necessary to contact that peer. The information for a given peer is updated whenever
a search request or reply is received from that peer by the query routing service. Furthermore, the
peer selection service indicates which of the known peers are currently nearby, i.e. which are physical
neighbours to the current peer. This can be used by the query routing service to improve the selection
of the peer or peers to which a search request should be forwarded.
3.3.7 Peer Management Service
The configuration of the middleware’s various services is handled by the peer management service.
This service maintains a global list of configuration parameters, for example, the file system path to the
storage service’s document databases or the network interface used for sending and receiving search
requests and search replies. These configuration parameters can be specified in a configuration file,
which is loaded on middleware start-up, and can, where applicable, be changed during runtime. As
such, other services consult the peer management service in order to obtain the parameters necessary
for their operation and the definition of their behaviour.
Beyond managing the middleware configuration, the peer management service also exposes statisti-
cal information, gathered from the various middleware services, through an embedded SNMP agent.
This allows other peers, and eventually static nodes configured as Network Management Systems
(NMS), to query the current peer’s middleware, via the SNMP protocol, in order to determine, for ex-
ample, its performance in responding to search queries or the number of locally stored documents. A
complete list of the variables exported by the agent, and which can be queried by remote peers, is
provided in Appendix B along with the structure of the corresponding SNMP Management Information
Base (MIB). The middleware services which provide information via the SNMP agent include: the query
routing service, the storage service, the service registration service and the peer selection service.
3.3.8 Middleware Communication Service
The middleware communication service is responsible for providing an interface between applications
and the middleware, in addition to enabling the communication between instances of the middleware
running on different peers. The former is accomplished by presenting applications with an object oriented
API. This API, which can be consulted in its entirety in Appendix C, exposes the functionality provided by
the middleware storage service, query routing service and service registration service to the application.
It allows an application to create, remove and share its document database as well as add, remove,
51
update and search for documents contained in said database. Documents can be searched either locally
or using the network by supplying a query string conforming to the syntax described in Section 3.3.3. If
a search is to be performed locally, the storage service is consulted, otherwise the query routing service
is used. Finally, applications can register services and remove existing registrations through the API.
The middleware also requires the ability to contact other middleware instances in order to obtain
documents matching a given query and, in the case of large result sets, the query results themselves.
In view of this requirement, the middleware communication service provides a RESTful web service
using HTTP, the peer communication interface, which can be accessed by remote peers. Three types of
resources can be obtained through this interface: documents, files and result sets. Any document stored
by the storage service can be retrieved given its unique identifier. File-backed documents, in addition to
the document itself, also have a corresponding file in the peer’s file-system, which can be retrieved using
the unique identifier of any file-backed document which refers to that file. Large result sets, i.e. those
which could not be transported in a search reply due to their size, can be retrieved using the unique
result identifier contained in such search replies.
The web service can be easily extended to support additional resources, and operations on such
resources, in order to address further middleware communication needs. The use of HTTP as the basis
for this service satisfies an important requirement for transferring both documents and files, i.e., transfers
can be resumed if interrupted. Consider a file with multiple source peers, if the source peer from which
the file is being transferred were to fail, another source peer could be chosen to retrieve the remainder of
the file. This is particularly relevant in the context of a MANET, where changes in the underlying network
paths are common and can result in the breakage of established HTTP connections. Partial resource
transfer is achieved through the use of the content range header as defined in HTTP/1.1 [45].
52
4Implementation
Contents
4.1 Extending HWMP . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55
4.2 P2P-HWMP . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 56
4.3 Middleware Daemon . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 58
4.4 API . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62
53
An implementation of the middleware described in the previous chapter was developed for the
GNU/Linux operating system. Current versions of the Linux kernel, 2.6.26 and later, include open80211s
[46], the first and foremost open source implementation of the IEEE 802.11s draft, namely of its default
routing protocol, HWMP, and link metric, the airtime link metric. open80211s is part of the new Linux
IEEE 802.11 stack, mac80211, and allows nodes with compatible hardware to establish infrastructure-
less wireless networks such as WMNs and MANETs. A modified version of this HWMP implementation,
included with version 2.6.29 of the Linux kernel, serves as a basis for the developed middleware.
As a result of the in-kernel nature of open80211s, the middleware is comprised of both kernelspace
and userspace components. P2P-HWMP, which is tightly coupled to HWMP, is implemented as a load-
able kernel module (LKM) while the remaining middleware components are implemented in userspace.
Kernel-space components are therefore limited to those which must interact directly with HWMP, i.e.,
P2P-HWMP, so as to limit the amount of code running with elevated privileges as well as to reduce
implementation complexity and avoid compromising the stability of the Linux kernel. The inclusion of fur-
ther middleware functionality in the kernel would not only suffer from restrictions imposed by the kernel
API but also from a lack of flexibility and portability.
The majority of the userspace middleware components are implemented by a daemon which runs as
a background process, independent of the P2P applications. This approach was chosen over the use
of a library, called by each application during its execution, in order to enable persistent functionality,
TCP/IP
Application
MIddleware (Daemon)
mac80211
P2P-HWMP (LKM)
HWMP (open80211s)
Linux Wireless Drivers
cfg80211
Kernel/Userspace boundary
API
Figure 4.1: Peer software stack
54
such as the discoverability of services or remote monitoring, despite the lack of currently running appli-
cations. Furthermore, the use of a single middleware instance reduces the memory footprint required
by applications and improves their start-up time. Given that the storage service stores and manages all
shared application content, an application need not be running in order for its content to be searchable
and retrievable.
Lastly, P2P applications use the middleware API in order to interact with the middleware. This API
corresponds to a thin library which employs an appropriate inter-process communication (IPC) mech-
anism to contact the middleware daemon. The API hides the fact that the remaining userspace mid-
dleware components are running in a separate process as well as the protocol used to interact with
the middleware daemon. Figure 4.1 illustrates the software stack at each peer. The implementation of
the middleware based solution consists of four distinct parts: the modification of the selected HWMP
implementation, the P2P-HWMP kernel module, the middleware daemon and the API library. Each part
is described in further detail in the following sections.
4.1 Extending HWMP
Extending the selected HWMP implementation encompassed the enhancement of the path selection
frame transmission function, the implementation of the HWMP vendor registry and the modification of
the code path used to process received path selection frames. Path selection frames refer to the frames
used by HWMP during the path discovery process, namely PREQ and PREP frames. HWMP is included
in the Linux mac80211 wireless stack which can be compiled into the kernel or as a separate loadable
kernel module. The latter option was used to facilitate development and testing of the aforementioned
functionality.
First off, the function used by HWMP to send new path selection frames, mesh path sel frame tx,
was extended in order to support an additional argument: a buffer containing a set of IEEE 802.11
vendor-specific information elements. This allows the inclusion of additional payload alongside path
selection messages. This function is used exclusively by the HWMP implementation and is therefore
only available within the name-space of mac80211, i.e, other kernel modules, such as P2P-HWMP,
are unable to access this functionality in order to transmit new path selection frames. As a result, the
mesh path sel frame tx function was exported using EXPORT SYMBOL, allowing it to be used anywhere
within the kernel. The caller can specify the type of path selection message, i.e. PREQ, PREP or path
error (PERR), as well as the value of each field in such message, including the TTL, destination address,
path lifetime, sequence number, among others.
The mesh vendor registry, as described in Section 3.3.1, was implemented so as to allow received
path selection frames to be handled outside of mac80211. The mesh vendor registry consists of a table
55
which stores for each registered entity, such as another kernel module, a set of function pointers, one
for each type of path selection frame the entity is interested in processing. Each entity is identified by a
vendor OUI which must coincide with the OUI used in that entity’s vendor-specific information elements.
The OUIs of the vendor-specific information elements contained in received path selection frames are
used to determine the entity and corresponding handler. Two new functions are exported by mac80211,
mesh vendor register and mesh vendor unregister, which can be used anywhere within the kernel
to, respectively, register and unregister an external entity and its handler functions.
The code path followed by received path selection frames, within mac80211, was changed in order
to call the HWMP vendor registry before any forwarding action is taken, or any reply generated, by
HWMP. Each received path selection frame is scanned for vendor-specific information elements whose
OUIs match the OUI of an entity registered with the vendor registry. If such a match occurs, a reference
to the frame, as well as the contents of the matching information element, are passed when calling
the corresponding handler function registered for that frame type. Recall that handlers can be either
active or passive, this being determined by their return value. If at least one active handler was called
for given frame, no further action is taken by HWMP once all vendor-specific information elements for
that frame have been processed. This assumes that the active handler is responsible for forwarding or
replying to the received path selection frame. If on the other hand, no active handler was called for a
given frame, control is passed back to HWMP which will continue processing the frame and follow the
standard forwarding behaviour.
4.2 P2P-HWMP
Since the selected HWMP implementation resides in the Linux kernel, P2P-HWMP, which is tightly
coupled with HWMP, is implemented as a loadable kernel module. P2P-HWMP interacts with HWMP by
calling the path selection frame transmission function and the registration and de-registration functions
of the vendor registry. P2P-HWMP also registers a set of handlers with HWMP in order to be notified of
received frames containing its vendor-specific information elements.
Communication with userspace middleware components, namely the middleware daemon, is achieved
through the use of netlink. Netlink is a Linux IPC mechanism used to exchange messages between the
kernel and userspace processes. For the former, an internal kernel API is provided while for the later,
the standard operating system socket interface is used with the AF NETLINK protocol family. Netlink sock-
ets offer a familiar programming interface for application developers and are similar to UNIX datagram
sockets in that both are datagram oriented, limited to the boundaries of the current host and address
processes rather than other hosts. Netlink is used extensively by recent versions of the Linux networking
stack, including mac80211, for exchanging networking information between the kernel and userspace
56
networking utilities. Alternative userspace to kernelspace interfaces, such as system calls or ioctl, do
not provide the flexibility offered by netlink, namely the ability to asynchronously notify the middleware
daemon of received search requests and replies.
On start-up, P2P-HWMP creates a new netlink socket, with a specific protocol, which listens for any
messages received from userspace. The messages supported by the implemented protocol, used for
the interaction between P2P-HWMP and the middleware daemon, are detailed in Appendix D. When
a middleware registration message is received from userspace, P2P-HWMP records the process iden-
tifier (PID) of the sending process and registers both an active PREQ handler and a passive PREP
handler with the HWMP vendor registry. An unassigned OUI, 99:99:99, is used to identify P2P-HWMP,
which can now be notified by HWMP of any frames containing vendor-specific information elements
which match this OUI. The registration with the HWMP vendor registry is maintained only while the mid-
dleware daemon is also registered with P2P-HWMP. If the connection to the daemon is lost or if the
daemon decides to unregister itself with P2P-HWMP, both handlers will be unregistered. This behaviour
guarantees that, if the middleware daemon is not running, P2P-HWMP will not be called and will not
interfere with the processing and forwarding of PREQ and PREP frames sent by other peers.
When a new search request is received from the middleware daemon, P2P-HWMP segments this
request into a set of vendor-specific information elements identified by P2P-HWMP’s OUI. The size
of the search request is limited to 1000 bytes in order to guarantee that, with the addition of the re-
maining management information and MAC header, the enclosing frame does not exceed the network
interface’s MTU, circa 1500 bytes for a default Linux network interface. The path selection frame trans-
mission function, exported by mac80211, is used to create and send a new PREQ frame, containing the
constructed information elements, to the neighbour nodes specified by the middleware daemon. This
information is included alongside the received search request and is comprised of a set of one or more
node addresses. If the broadcast address, ff:ff:ff:ff:ff:ff, is specified, the frame will be sent to
all neighbour nodes. Another peer, upon receiving this frame, will scan the enclosed vendor-specific
information elements and call P2P-HWMP’s PREQ handler for each information element matching P2P-
HWMP’s OUI. The PREQ handler is passed a reference to the PREQ frame and the contents of the
corresponding information element. These information elements are used by P2P-HWMP to reassem-
ble the original search request which is in turn sent to the middleware daemon, via netlink, using a search
request notification message. The PREQ frame containing the search request is stored by P2P-HWMP
until either a search reply or a forwarding decision is received from the middleware daemon.
After processing the search request, the middleware daemon will communicate its forwarding deci-
sion to P2P-HWMP using a forwarding decision message. This message instructs P2P-HWMP on the
forwarding action it should take for the stored PREQ frame. The frame can either be forwarded, as de-
scribed previously for the requesting peer, or dropped altogether. In addition to a forwarding decision for
57
a given search request, the middleware daemon can also provide a search reply for transmission back to
the requesting peer. In this case, the stored PREQ frame is used to determine some of the fields, such
as the destination address, for the PREP frame which will be created to transport the search reply. The
search reply is segmented in a manner analogous to that described for the search request, inserted into
a new PREP frame and sent to the next hop on the path back to the requesting peer. The stored PREQ
frame can now be discarded. Once received at the requesting peer, the reassembled search reply is
sent to the middleware daemon using a search reply notification message.
Frequently, when the requesting peer receives a reply for a search request, the P2P application will
attempt to contact the replying peer in order to exchange further information. This is achieved through
the use of a transport protocol, such as TCP or UDP, which relies on the network-layer to address the
replying peer. Since HWMP addresses peers at the link-layer, the network address of the replying peer
is not directly available to the requesting peer. Even with knowledge of the network address, the normal
behaviour of the requesting peer would be to determine the mapping from network address to link-layer
address through the use of the Address Resolution Protocol (ARP), for IPv4 network addresses, or
Neighbour Discovery Protocol (NDP), for IPv6 network addresses. In both cases, this entails flooding
the network with an address resolution request, in the case of ARP, and a neighbour solicitation, in
the case of NDP, to which the target node, which matches the network address, would reply. This
additional broadcast traffic further degrades the performance of the MANET. P2P-HWMP provides an
optimisation to avoid executing ARP or NDP if the requesting peer decides to contact the replying peer
immediately after receiving a search reply. Both search requests and search replies contain the network
address of the originating peer in their respective headers, these headers are scanned by P2P-HWMP
upon reception and used to create a new entry, mapping the originating peer’s network address to its
link-layer address, in the kernel ARP or NDP table. Thus, if the application were to establish a TCP
connection to the replying peer, using its IPv4 network address, the kernel would already have an entry
in the ARP cache mapping the peer’s IPv4 address to its link-layer address without the need to generate
further network traffic to discover such mapping.
4.3 Middleware Daemon
The core functionality of the middleware is implemented by the middleware daemon, namely the
communication, storage, query routing, service registration, peer selection and peer management ser-
vices. As mentioned previously, this daemon consists of a background process, typically executed on
system start-up, which is contacted by P2P applications through the middleware API.
The middleware daemon was developed using Python [47], a general purpose dynamic object-
oriented programming language which emphasises code readability and maintainability through clear
58
syntax. Python was chosen due to its ease in rapidly prototyping large applications as well as its com-
prehensive standard library, which implements most common internet protocols and data formats. Given
the middleware deamon’s multi-protocol nature and its need to interact with multiple external processes
simultaneously, both locally and at remote peers, an event-driven network programming framework,
Twisted [48], was chosen as a basis for the implementation. Twisted is particularly suited for complex
network intensive applications and includes support for developing clients and servers using a variety of
protocols, namely TCP, UDP, HTTP, Telnet and XML-RPC (XML Remote Procedure Call), among oth-
ers. Since multiple concurrent network and IPC communication links can be integrated with the Twisted
event-loop, the use of threading and associated synchronisation primitives can be avoided, simplifying
the access to shared data structures.
The middleware daemon was implemented as a Twisted application. A Twisted application is charac-
terised by the use of the Twisted Application Infrastructure to handle application startup and shutdown,
logging, forking to background, selection of the event-loop, or reactor in Twisted terminology, among
other common tasks. This reduces the amount of code required to setup such functionality in the actual
application. The central concept of a Twisted application is the Application object, representing the ap-
plication. This object acts as a container for Twisted Service objects, i.e., entities within the application
which can be started and stopped. Twisted already provides several services, such as, web servers,
Telnet servers, TCP servers and clients, which can be easily extended and on top of which custom pro-
tocols can be constructed. A Twisted application can contain multiple services organised hierarchically.
Each of the previously mentioned middleware components, such as the storage service, which compose
the middleware daemon, are implemented as Twisted services.
Wherever possible, an existing Twisted service was used as a basis for a given middleware service.
The dependencies between the various services are presented in Figure 4.2, where a directed edge rep-
resents the invocation of methods provided by the target service. As expected, all services rely on the
management service in order to obtain configuration information, necessary to configure their respec-
tive behaviours. Moreover, the middleware communication service acts as a proxy between local P2P
applications, or remote peers, and the several services provided by the middleware, such as, storage or
query routing. Beyond implementing the functionality described in Chapter 3, some of the middleware
services have additional implementation details which are worth mentioning.
The storage service relies on the Xapian [49] open source search engine library in order to enable the
indexing and searching of P2P application documents. Xapian provides full text searching capability for
the indexed content, along with, structured boolean queries, unicode support, wildcard searching, large
database support (greater than 2GB) and simultaneous searching and indexing. Despite being written in
C++, bindings for a variety of programming languages, including Python, are available for accessing the
Xapian core. Rather than use the interface provided by the Python Xapian bindings, which deals directly
59
Storage
Management
Service Registration
Query Routing
Peer Selection
Middleware Communication
Figure 4.2: Dependencies between middleware services provided by the middleware daemon
with low-level information retrieval data structures, a wrapper module, Xappy [50], was used instead.
The Xappy module provides a higher-level interface to Xapian, allowing the definition of a field structure,
including the type of information stored at each field, for a given document. This field structure is used to
specify how the document is indexed, based on its fields, and in constructing and performing searches.
A new Xappy database is created in the peer’s filesystem for each individual P2P application. Once a
document has been indexed by Xappy, it is stored in this database alongside the document index. The
Xappy database provides the underlying storage for an application’s document repository. In addition to
the documents themselves, this database is also used for storing metadata, such as, the password for
the document repository, the list of fields which should be indexed for newly added documents and the
list of fields which should be included in search reply messages.
A new datagram-oriented Twisted transport, using netlink sockets, had to be developed in order to
support the interaction between the query routing service and P2P-HWMP. For the purpose of demon-
strating the viability of the designed middleware and comparing it to similar solutions, a query flooding
protocol, with a configurable TTL, was chosen as the query routing protocol for the query routing service.
A search request received at a given peer is therefore broadcast to all neighbour nodes. This is done
regardless of whether or not the peer replies to the enclosed query. Despite the implementation of a
single query routing protocol, it is important to note that the required infrastructure is in place to support
further protocols, such as those described in Section 3.3.4. The forwarding decision message sent to
P2P-HWMP, as well as P2P-HWMP itself, already allow the specification of the addresses the search
request, and corresponding PREQ frame, should be forwarded to. The format of the search messages,
60
http://NODE_ADDRESS:PORT/document/APPLICATION_ID/DOCUMENT_ID
http://NODE_ADDRESS:PORT/document/APPLICATION_ID/DOCUMENT_ID?type=file
http://NODE_ADDRESS:PORT/result/RESULT_ID
Figure 4.3: Resource URLs provided by the peer communication interface
.
namely search requests and search replies, exchanged between the query routing services at different
peers, is detailed in Appendix A.
In order to perform source peer selection, the peer selection service requires a copy of the mesh
path table containing the metrics for the paths discovered by HWMP. This mesh path table is stored,
by mac80211, as an in-kernel memory structure and is therefore not easily available to userspace pro-
cesses. The command-line networking utility, iw, can be used to query the kernel and extract such
information, but, this requires spawning an external process, at regular intervals, to allow the peer se-
lection service to keep an updated copy of the mesh path table. Since mac80211, and the networking
information it stores, can also be queried through netlink, via the nl80211 interface, the peer selection
service periodically polls the kernel in order to obtain an updated copy of the mesh path table. This is
achieved by implementing the protocol necessary to interact with nl80211, and extract the mesh path
table, on top of the previously mentioned Twisted netlink transport.
The web server implementation included with Twisted, and provided as a Twisted service, was used
as a basis for the middleware communication service. This implementation includes support for the
HTTP/1.1 protocol, thus allowing partial retrieval of resources by taking advantage of the Content-Range
header. Two interfaces are provided, on top of HTTP, by the middleware communication service, one
for use by remote middleware instances and another for use by the API library. The first interface,
the peer communication interface, which consists in a RESTful web service, is used for transferring
documents, files associated with file-backed documents and result sets. These resources are retrieved
using the HTTP GET method on the appropriate URL, as shown in Figure 4.3. The first URL, from top
to bottom, is used for retrieving a document based on its unique identifier and the unique identifier of
its application. The latter determines which document repository to retrieve the document from. If the
document is a file-backed document, the associated file, residing in the local filesystem, can be retrieved
by specifying the type=file parameter as shown in the second URL. Finally, the third URL is used for
obtaining a previously cached result set based on its unique identifier. While the representation of a file
consists of its actual content, documents and result sets require an appropriate representation in order
to be transferred. Javascript Object Notation (JSON) was used to encode and represent both types of
resources, enabling their reconstruction at the requesting peer. This JSON representation of the result
set is also used when transferring result sets via the query routing service’s search reply messages.
61
The second interface provided by the middleware communication service, the application interface,
consists in an XML-RPC web service. This web service exposes the functionality provided by the stor-
age, service registration and query routing services, to the API library. While the peer communication
interface is available to remote peers, the application interface is restricted to processes running on the
local peer and intended for internal use by the middleware at that peer. This interface provides an IPC
mechanism between the API library and the middleware daemon in a manner which is transparent to
the P2P applications. These interact exclusively with the API library as discussed in Section 4.4.
Lastly, the SNMP agent functionality, provided by the peer management service relies on Twisted-
SNMP [51], an SNMP protocol implementation for Twisted based on PySNMP, which is not included in
the base Twisted distribution. The SNMP agent provides a read-only MIB, whose content is detailed in
Appendix B, and can be queried by any SNMPv2 compatible client. The OIDs described in the MIB are
mapped to methods at the corresponding middleware service, responsible for calculating and returning
the requested value.
4.4 API
The API library is the only middleware component a P2P application interacts with directly. This
library, which provides an object-oriented API, is also written in Python, allowing application developers
to quickly develop applications with minimal effort. Since most middleware functionality is implemented
by the middleware daemon, the API library is merely a collection of classes for creating new documents
and for wrapping the methods provided by the previously mentioned XML-RPC interface. As a result,
the idiosyncrasies of the communication between the application process and the middleware daemon
process are hidden from the application, resulting in a more flexible interface.
The API itself consists of two Python modules, middleware.types and middleware.api, which must
be imported by applications wishing to take advantage of the middleware. The middleware.types mod-
ule provides a set of classes: Document, File and ServiceDescription, which are used to create
documents, file-backed documents and service description documents respectively. Theses classes
present an interface similar to a Python dictionary for getting and setting document fields. In addition to
this module, the middleware.api module provides a single class, Middleware, which represents a con-
nection to the middleware. This class is used by an application for all interactions with the middleware.
It allows the manipulation of an application’s document repository, namely creating a new repository;
adding, removing, updating and obtaining local documents; removing a repository; setting a repository
password and enabling and disabling sharing for that repository. It also allows the registration and de-
registration of services provided by the application and the ability to specify a query for use in searching
the network for matching documents. Appendix C illustrates the structure of both modules, in addition to
62
providing the source code for an example application using the middleware API. This example applica-
tion searches the network for peers running the middleware agent service and prints the URLs contained
in the service description documents for the discovered services.
Since some of the methods of the Middleware class may block while waiting for results from the
middleware, due to the synchronous nature of XML-RPC invocation, e.g. the method used to search
for remote documents, an asynchronous version of this class is provided in addition to the synchronous
implementation. This asynchronous version is built using Twisted, whereby methods that would block
return immediately. These methods return a special object, a Deferred in Twisted terminology, to which
callbacks can be attached. The callbacks are executed when the actual results of the invocation are
returned. Thus, the API allows for both synchronous and asynchronous call semantics.
63
5Testing and Evaluation
Contents
5.1 Testbed . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 65
5.2 Test Metrics and Scenarios . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 66
5.3 Experimental Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 68
64
In order to validate the implemented middleware solution and evaluate its performance, a set of func-
tional and performance tests were conducted using a MANET testbed consisting of six nodes organised
according to a specific network topology. The primary focus of these tests was the search functionality
offered by the middleware, and the resulting query routing, as this, along with source peer selection,
constitutes the core functionality of the middleware. In the context of the performance tests, the middle-
ware was also compared to a traditional layered solution, using Gnutella, which was not designed for
use in a MANET environment.
5.1 Testbed
The testbed consisted of six nodes equipped with IEEE 802.11 network interfaces forming a multi-
hop MANET using IEEE 802.11s and its default routing protocol, HWMP. The hardware and software
configuration, which is identical for all nodes, is presented in Table 5.1. The hardware used is represen-
tative of the sort of scenarios where the middleware may be employed and therefore provides a good
indication of real world performance. Unless explicitly noted, all test scenarios assume, in addition to the
software listed, that the middleware is installed and running on each node, i.e., that each node is also a
peer.
All nodes in the testbed were connected to a wired LAN, in addition to the MANET, in order to
facilitate the configuration and monitoring of the testbed. This LAN was used to remotely manage all
nodes without interfering with the traffic exchanged via the MANET and for providing access to external
services such as time synchronisation.
System ASUS Eee PC 900A
Processor Intel Atom N270 1.6 GHz
RAM 1 GB DDR2-400
Wireless NIC Atheros AR2425 802.11abg (ath5k)
Operating System Debian GNU/Linux 5.0.1 (”Lenny”)
Software Linux Kernel 2.6.29 (with HWMP extensions)
Python 2.5.4
Twisted 8.1.0
TwistedSNMP 0.3.13
Xapian 1.0.7
Xappy 0.5
gtk-gnutella 0.96.7u-16948
Table 5.1: Testbed node configuration
65
3
1 4
5
2
6
Figure 5.1: Testbed network topology
The topology of the MANET is presented in Figure 5.1, where the edges of the graph represent
bidirectional links. This topology was chosen among the set of possible topologies since it satisfies the
following criteria: it represents a multi-hop network; multiple paths, of differing hop counts, exist between
two nodes and, despite the average node degree, three in the case of the selected topology, the failure
of a single node splits the network into two separate partitions. Thus, the selected topology allows for
the evaluation of both the query routing functionality of the middleware, as well as, the ability to select
an optimal source peer when multiple source peers exist for a given document.
As all nodes were placed near each other, with line-of-sight between each pair of nodes, node visi-
bility, i.e. the ability for a node to detect the presence of another node and establish a link to that node,
was enforced through the appropriate configuration of mac80211. Any frame received from a node to
which the receiving node does not have an edge, according to the graph representing the topology in
Figure 5.1, is dropped at the link-layer, before being passed to HWMP. Therefore, all discovered network
paths conformed to the defined network topology.
5.2 Test Metrics and Scenarios
The assessment of the implemented middleware was achieved through both functional and perfor-
mance testing. On the one hand, functional testing was used to verify the correct operation of the
middleware, namely its ability to coexist with legacy nodes, select source peers and search for, and en-
able the discovery of, documents and services on the network. Performance testing, on the other hand,
was used to determine the delay introduced by various components of the middleware software stack, to
compare the efficiency of the middleware with Gnutella and to evaluate the middleware’s query routing
performance. The terms query and search request are henceforth used interchangeably to refer to the
message containing the application supplied query string.
66
The metrics used consist of the query forwarding delay, query success rate, overall traffic and query
processing time. The query forwarding delay represents the time elapsed between the transmission
of a received search request, by P2P-HWMP, to the query routing service and the reception of the
forwarding decision for that search request. This metric is used to evaluate the overhead introduced by
taking forwarding decisions regarding a search request in userspace. The query success rate is defined
as the fraction of queries for which at least one reply was received over the number of queries sent. The
query success rate only includes queries for which at least one matching document exists at another
peer in the network. The overall traffic, both in terms of total size and number of messages, refers
to the network traffic which is generated as a direct consequence of a search request and includes
the overhead introduced by the IEEE 802.11 MAC, HWMP path discovery, ARP address resolution,
transport and application-layer protocols. Lastly, the query processing time refers to the wall clock time
spent processing a given search request at an intermediate peer, i.e, the time elapsed between receiving
a PREQ frame and forwarding it to the next hop. This processing time is divided according to the
middleware components involved, namely P2P-HWMP, netlink communication, the query routing service
and the storage service. Although similar to the query forwarding delay metric, the query processing time
is more encompassing and provides insight into which middleware components have the greatest impact
on the time required for a search request to traverse a peer.
Four scenarios were created for the purpose of testing the middleware. In the first scenario, a sim-
ple file-sharing application is run at each peer. This application uses the middleware API to create a
document repository and creates a file-backed document for each file in a specified directory. These
file-backed documents are then added to the repository, which is marked as shared, enabling other
peers to discover the files contained in the directory. Each peer shares a directory containing a few
dozen files. To test the correct operation of the middleware search functionality, a peer sends a search
request for a specific file shared by another peer in the network such that at least two intermediate
peers exist between the requesting peer and the replying peer. This constraint is used to exercise query
forwarding over multiple hops. The expected result consists in the reception of a search reply for that
request. Beyond validating the search functionality, this scenario is used to obtain experimental perfor-
mance results using the query forwarding delay, overall traffic, query success rate and query processing
time metrics.
Unlike the first scenario, the following scenarios are dedicated exclusively to functional testing. The
second scenario is a variation of the first, where the intermediate nodes between the requesting peer
and the replying peer do not run the middleware software stack, i.e., the intermediate nodes are legacy
nodes as opposed to the P2P nodes in the first scenario. This scenario is used to validate the correct
operation of the middleware search functionality in the presence of legacy nodes. The expected result
is analogous to that defined for the first scenario.
67
The third scenario is used to test the source peer selection functionality provided by the middleware.
This scenario is also based on the first but, rather than have a single source peer for the requested
document, two source peers, A and B, are available. Although both source peers are chosen to be the
same number of hops away from the requesting peer, peer A is setup adjacent to the requesting peer
while peer B is located in another room. Before executing the test, the throughput achieved on the path
between the requesting peer and each of the source peers is measured to ensure that it is lower in the
case of the path to peer B. As a result of this scenario, peer A should be selected by the middleware as
the first source peer to be contacted, as opposed to peer B, upon the reception of two search replies,
one from each source peer.
Lastly, the fourth scenario verifies the service discovery functionality of the middleware. A given
peer initiates a new search request for a specific service, the middleware communication service, which
is offered by every peer in the network. The expected result is the reception of a search reply from
each peer in the network containing a service description document for the middleware communication
service.
5.3 Experimental Results
The four test scenarios described in the previous section were carried out in order to validate the
operation of the middleware. For each scenario, the results obtained for the transmitted query corre-
sponded to those expected for that scenario. In the case of the third scenario, the source peer selection
functionality was able to select the source peer to which the requesting peer had the lowest value of
the path metric. This was confirmed by consulting the kernel mesh path table. Following the functional
testing, the first scenario was used to obtain experimental results for the query forwarding delay, query
success rate, overall traffic and query processing time metrics. No background traffic was present in the
MANET during each measurement, with the exception of the traffic required by the IEEE 802.11 MAC,
namely periodic beacon transmissions announcing the presence of each node.
The query forwarding delay at a given peer was measured for an increasing query load, i.e. the
number of queries received by that peer during a given unit of time, in order to determine the impact
of the query load on the forwarding delay introduced by the userspace middleware components. The
query forwarding delay was measured through P2P-HWMP, which calculated the time elapsed between
transmitting a received search request, containing a query, to the middleware daemon and receiving
a forwarding decision for that search request. Node 1 was selected as the query generator and node
4 was used to measure the query forwarding delay for the queries received from the query generator.
The query generator was configured to generate queries at different rates: 1, 10, 100 and 1000 queries
per second corresponding to an inter-query spacing of 1000, 100, 10 and 1 milliseconds, respectively.
68
0
7.5
15.0
22.5
30.0
1 10 100 1000
Qu
ery
fo
rwa
rdin
g d
ela
y (
ms)
Queries per second
Without reply With reply
Figure 5.2: Query forwarding delay for increasing query load
Measurements were performed for two cases: when the generated queries match a document stored at
the peer performing the measurements, thus generating a reply, and when no matching documents exist.
Figure 5.2 shows the results obtained for both cases during a 10 second period. For low query loads, i.e.
up to 100 queries per second, the forwarding peer exhibits an almost constant query forwarding delay of
approximately 10.3 ms if the query matches a document stored by the peer and 7.5 ms otherwise. As
expected, the query forwarding delay is higher in the case of the former when compared to the latter,
circa 27%, since the middleware storage service must not only consult the Xapian index to determine
if matching documents exist but also extract those documents from the underlying storage in order
to return the results to the query routing service. Only then can the query routing service deliver a
forwarding decision to P2P-HWMP. If the query load is higher than 100 queries per second, the query
forwarding delay begins to increase since the middleware is unable to finish processing and forwarding
the current query before receiving the next query.
Although the query forwarding delay can be used to analyse the overhead introduced by the userspace
middleware components in forwarding a query, it does not provide a complete overview of the time re-
quired for a query to traverse the middleware at a given peer, i.e., the query processing time. The query
processing time at a given peer is composed of the wall-clock time spent by each middleware compo-
nent, namely P2P-HWMP, netlink communication, the query routing service and the storage service, in
processing a received query. Time-stamps were generated at the handoff points between interacting
components, for example, when the query routing service calls the storage service or when P2P-HWMP
sends a message via the netlink socket. The processing time spent by each component was then
69
0 2.50 5.00 7.50 10.00
Processing time (ms)
P2P-HWMP NetlinkMiddleware (routing service) Middleware (storage service)
Figure 5.3: Query processing time
calculated based on the generated time-stamps.
Figure 5.3 presents the results of measuring the query processing time at a given peer, node 4, for
a received query matching a stored document. Both P2P-HWMP and netlink communication represent
a negligible contribution, 0.78 ms, to the overall processing time. Both these components are executed
in a kernel interrupt handler and therefore do not suffer from scheduling delays. Furthermore, these
components carry out only a limited set of operations on the received search request. The bulk of
the processing time, 8.6 ms, is split between the query routing service, where the search request is
parsed, and the storage service, where the query is resolved by consulting a document repository.
These services represent an opportunity for further optimisation with regard to the code path followed
in order to forward a search request. Furthermore, the priority of the middleware daemon process,
where both services are executed, could be increased to reduce the impact of operating system process
scheduling delays. In spite of this, the time taken by a query to traverse the middleware at a given peer,
9.38 ms, is in line with that estimated for the one-hop packet traversal time for traditional MANET routing
protocols. For AODV, the estimated packet traversal time, including the time required to transmit the
packet to the next hop and any queuing delays, is 40 ms [26].
The query success rate was measured at a given peer to determine how the ability to resolve a query
is affected by increasing query load. Node 1 was chosen to generate queries in a manner analogous
to that described for the query forwarding delay metric. All of the generated queries match at least
one document stored by another peer, node 5. For each query sent, the generating peer waits until a
reply is received or the query times out. The query success rate is then calculated as the number of
queries for which a reply was received over the number of queries sent. The measurements obtained
by the generating peer are presented in Figure 5.4. For query generation rates less than 10 queries
70
0
25
50
75
100
1 10 100 1000
Query
success r
ate
(%
)
Queries per second
Figure 5.4: Query success rate for an increasing query load
per second, the generating peer is able to resolve, on average, 99.5% of the transmitted queries as the
resulting query routing traffic, especially at node 4, is insufficient to cause network congestion. In this
case, the loss of PREQ frames containing search requests and PREP frames containing search replies
is negligible. The query success rate begins to decrease for query generation rates greater than 10
queries per second. Despite this, 76.8% of the queries are resolved in the case of 1000 queries per
second.
A comparison between the middleware and an off-the-shelf solution consisting of Gnutella, TCP/IP
and HWMP was used to analyse the efficiency of the implemented middleware solution with respect to
the overall traffic generated as a result of a single query, i.e., the volume of unicast and broadcast traffic
generated, both in terms of the number of frames as well as the total size. gtk-gnutella was chosen as
the Gnutella client and installed on every node in the testbed alongside the middleware. gtk-gnutella is a
popular Gnutella client for Unix-like operating systems written in C with a GTK+ graphical user interface
and the ability to run in the background, as a daemon, as well as a command-line interface with limited
functionality. The latest version of the Gnutella protocol, version 0.6, is supported by gtk-gnutella as well
as many proposed extensions including IPv6 and Transport Layer Security (TLS) support. gtk-gnutella
is also highly configurable such that most aspects related to the behaviour of the Gnutella protocol can
be customized. For this test, gtk-gnutella was configured in ultra-peer mode, without any leaf nodes,
in order to reflect the query flooding nature of Gnutella and the lack of node hierarchy inherent in a
MANET. Both TLS and compression support were disabled for Gnutella overlay connections to provide
a fair comparison with the middleware which does not support this functionality. Furthermore, most
extensions, such as partial file sharing and swarming, were disabled.
71
0
37500
75000
112500
150000
Middleware Gnutella (warm) Gnutella (cold)
Tra
ffic
(b
yte
s)
MAC HWMPTCP OtherGnutella (Other) Gnutella (Ping/Pong)Gnutella (Query) MIddleware
Figure 5.5: Overall traffic (bytes) generated for a single query
An additional node was used to measure the overall traffic generated. This node was equipped with
an IEEE 802.11 network interface operating in monitor mode to capture all traffic exchanged between the
nodes participating in the MANET. Node 5 was configured to share a given file using both gtk-gnutella
and the P2P file-sharing application described in the previous section. At node 1, each application
was then used to send a query for this file. In the case of the gtk-gnutella, the query was routed
through the Gnutella overlay, constructed by randomly selecting the set of neighbouring peers for each
peer. The topology of the overlay was generated such that the average peer degree, i.e. the number
of neighbours, matched that of the underlying network topology presented in Figure 5.1. The set of
neighbours determined by the generated overlay topology were used to bootstrap each Gnutella peer.
Network traffic was captured during a 30 second interval and analysed using the Wireshark packet
analyser for a query sent according to three scenarios: using the file-sharing application developed
with the middleware API, henceforth known only as middleware, using gtk-gnutella with a previously
established overlay, henceforth known as Gnutella warm, and using gtk-gnutella while establishing the
overlay during the measurement interval, henceforth known as Gnutella cold. It is important to note that
in the case of the middleware scenario, the query results were embedded in the search reply message
sent to the requesting peer rather than being obtained out-of-band using the middleware communication
service.
The overall traffic generated as a result of the query, in terms of total size of the transmitted IEEE
802.11 frames, is displayed in Figure 5.5 for the three scenarios. The composition of the total size is
also shown, where each transmitted frame is classified according to its payload: IEEE 802.11 MAC
management frames, HWMP path selection frames (excluding those with middleware payload), TCP,
72
0
225
450
675
900
Middleware Gnutella (warm) Gnutella (cold)
Tra
ffic
(n
um
be
r o
f m
essa
ge
s)
MAC HWMPTCP OtherGnutella (Other) Gnutella (Ping/Pong)Gnutella (Query) MIddleware
Figure 5.6: Overall traffic (messages) generated for a single query
the various Gnutella protocol messages (query, query hit, ping, pong), frames containing middleware
search requests or search replies and finally other frames used, for example, for ARP address resolution
requests and responses. In the case of the middleware scenario, only two types of frames are generated:
IEEE 802.11 MAC management frames and HWMP path selection frames containing search requests
or search replies. The middleware scenario represents a 74% decrease in traffic when compared to
the Gnutella warm scenario and an 85% decrease when compared to the Gnutella cold scenario. The
overhead introduced by Gnutella is mainly due to the use of Gnutella Ping and Pong messages for
discovering other peers in the overlay and determining the status of connections to neighbouring peers.
Other factors contributing to the overhead include the successive HWMP route discoveries triggered in
order to maintain persistent TCP connections between overlay neighbours, as well as, the overhead
introduced by TCP itself including any packet retransmissions.
The 81% increase in traffic from the Gnutella warm to the Gnutella cold scenario is explained by the
need to establish the P2P overlay before sending the query. In order to establish the overlay, each peer
must initiate a TCP connection to each selected neighbour peer resulting in a TCP 3-way handshake per
connection and a Gnutella protocol handshake on top of the established connection where both peers
exchange the list of supported extensions. Since each peer’s ARP table is initially empty, ARP requests
must be broadcast to the network in order to discover the mapping between each peer’s network address
and its link-layer address. This is not an issue in the middleware scenario, where the transmission of
search requests and replies does not require the knowledge of a peer’s network address, and in the
Gnutella warm scenario since the ARP address resolution process has already been executed as a
73
0
37.5
75.0
112.5
150.0
MIddleware Gnutella (warm) Gnutella (cold)
Bro
adcast tr
affic
(m
essages)
Figure 5.7: Broadcast traffic (messages) generated for a single query
result of establishing the overlay.
Similar observations can be made for Figure 5.6 which presents the overall traffic generated in terms
of protocol messages. Additionally, the obtained results allow for a comparison between the number of
messages broadcast at the link-layer for each scenario. This is a factor in determining the efficiency of
a given solution, as broadcast traffic has a greater impact on network performance than unicast traffic.
Figure 5.7 shows the number of messages broadcast for the three scenarios, excluding IEEE 802.11
MAC beacon frames, as these are always present, with the same frequency, regardless of the scenario.
In the case of Gnutella, broadcasting a query using the overlay results in multiple messages unicast
at the link-layer, therefore, such virtual broadcasts are omitted. Despite this, the number of messages
broadcast for the Gnutella warm and Gnutella cold scenarios represent an increase of 2040% and
2900%, respectively, over the middleware scenario. While the middleware executes a single HWMP
path discovery process to propagate the query to all peers and receive the corresponding reply, Gnutella
uses static TCP connections between overlay peers. Keep-alive traffic, in other words, ping and pong
messages, are constantly exchanged over these connections. Since the paths discovered by HWMP
expire after 5s, each Gnutella peer will trigger a path discovery to each of its current overlay neighbours
in order to maintain the corresponding overlay connection. The PREQ frames which are broadcast as a
result of successive path discovery processes contribute to Gnutella’s poor result.
In short, one can easily conclude that establishing a P2P overlay network on top of a MANET rep-
resents a considerable maintenance overhead, even for an unstructured P2P system, when compared
to a P2P system which takes advantage of the underlying MANET routing protocol to transport queries
and the corresponding results, as is the case of the implemented middleware solution. Furthermore,
the use of static connections between peers in the overlay and keep alive messages to verify that such
74
connections are still valid is inefficient in a MANET as it triggers successive route discovery processes,
and consequent broadcast traffic, if a reactive routing protocol is used. Although not apparent due to the
size of the MANET testbed, the selection of overlay neighbours which are multiple physical hops away
will only exacerbate this problem.
75
6Conclusion
76
P2P networks and MANETs exhibit many similarities, including decentralisation, self-organisation
and a highly dynamic topology. A key issue in both networks is the process of discovering the requested
content or route efficiently in a decentralised manner and routing messages between nodes. As such,
it is clear that the P2P paradigm is particularly suited to enable content distribution, communication and
collaboration applications in a MANET environment. In spite of this, existing P2P systems, designed for
the Internet, do not exploit the synergies between P2P networks and MANETs since the P2P overlay
network is unaware of the topology of the underlying physical network. P2P application networks operate
at the application layer of the OSI reference model while MANETs operate at the network layer or data
link layer. Routing functionality is therefore present in both layers. On the one hand, application layer
routing is used by search queries to find a peer containing the desired information within the P2P overlay
network. On the other hand, network layer or data link layer routing is used to discover the route to
any given node. Logical node proximity differs from physical proximity, resulting in inefficient paths
between neighbours in the overlay and unnecessary network overhead due to the duplicate functionality
implemented at different layers in the network stack.
This thesis proposed a P2P middleware solution to facilitate the development of P2P applications
optimised for use in MANETs. This solution avoids duplicating the discovery and routing functionality
at different layers in the network stack by integrating the process of searching for application content
with the discovery of the network routes in the underlying MANET. Rather than use a P2P overlay
network consisting of a set of persistent connections setup between possibly distant peers in order to
route application layer search requests and replies, the MANET routing mechanisms are used instead.
HWMP was chosen as the MANET routing protocol due to its inclusion in the IEEE 802.11s draft, which
promises to become the de facto standard for the creation of multi-hop wireless mesh networks and
MANETs. Both HWMP path request and path reply messages were extended in order to transport,
respectively, application layer search requests and search replies. As a result, P2P applications can
submit new search requests to the network routing layer and be notified of the reception of new search
requests. When multiple sources are available for the same content, the middleware chooses a peer
based on the cost of the path to that peer, as given by HWMP’s path metric. The middleware therefore
provides applications with an optimised, network-aware, source peer selection service, in addition to the
standard search functionality.
An implementation of the proposed P2P middleware was developed for the GNU/Linux operating
system and based on the HWMP and airtime link metric implementations included with the Linux ker-
nel IEEE 802.11 stack, mac80211. As such, the middleware implementation was divided into both
kernelspace components, which interact directly with HWMP, and userspace components, which com-
prehend the middleware daemon, providing most of the middleware functionality, as well as an object-
oriented Python API to enable developers to easily develop P2P applications which take advantage
77
of the middleware. An appropriate communication channel, using netlink, supports the exchange of
information between both userspace and kernelspace middleware components.
The middleware implementation was evaluated experimentally through both functional and perfor-
mance tests executed on a small multi-hop MANET testbed consisting of nodes equipped with IEEE
802.11 network interfaces. These tests focused on the search functionality provided by the middleware,
and the resulting query routing. The results obtained for the various metrics demonstrated the reduced
overhead introduced by the middleware and the reduction in network resource usage when compared
to Gnutella, a traditional P2P system, layered on top of a MANET.
Several points remain to be addressed as part of future work. First of all, the query routing service
can be used to implement further query routing protocols, such as, random walk and expanding ring
search. The impact of these protocols on the quality of the paths discovered by HWMP, the delay in
resolving queries and the amount of traffic generated could be the focus of further study. Although
experimental performance results were obtained using a testbed, further testing, through simulation,
is required in order to obtain a more detailed characterisation of the performance of the middleware,
namely the impact of mobility and large node populations on the query success rate. Even though
the middleware provides the ability to discover services on the network, only the minimum amount of
information necessary to contact a service is included in the returned service description documents. An
existing service description language could be integrated with the middleware and used in constructing
service description documents. Finally, in spite of the lack of persistent TCP connections between
neighbouring peers, TCP is still used by the middleware in order to transfer both documents and, in
some cases, results from the replying peer. Further reliable transport protocols, designed for wireless
networks, could be used to address the performance issues introduced as a result of TCP operation in
MANETs.
78
Bibliography
[1] S. Jochen, Mobile Communications. Boston, MA, USA: Addison-Wesley Longman Publishing Co.,
Inc., 2003.
[2] R. Schollmeier, “A definition of peer-to-peer networking for the classification of peer-to-peer ar-
chitectures and applications,” in P2P ’01: Proceedings of the First International Conference on
Peer-to-Peer Computing (P2P’01), (Washington, DC, USA), p. 101, IEEE Computer Society, 2001.
[3] The Gnutella Developer Forum, Gnutella protocol specification 0.4, http://rfc-
gnutella.sourceforge.net/developer/stable/index.html.
[4] G. Ding and B. Bhargava, “Peer-to-peer file-sharing over mobile ad hoc networks,” in PERCOMW
’04: Proceedings of the Second IEEE Annual Conference on Pervasive Computing and
Communications Workshops, (Washington, DC, USA), p. 104, IEEE Computer Society, 2004.
[5] One Laptop Per Child project, http://www.laptop.org.
[6] A. Marques, F. Mira da Silva, and R. Rocha, “P2P over Mobile Ad-hoc Networks,” in Sensor, Mesh
and Ad Hoc Communications and Networks Workshops, 2009. SECON Workshops ’09. 6th Annual
IEEE Communications Society Conference on, pp. 1–3, June 2009.
[7] S. Androutsellis-Theotokis and D. Spinellis, “A survey of peer-to-peer content distribution technolo-
gies,” ACM Comput. Surv., vol. 36, no. 4, pp. 335–371, 2004.
[8] Jabber, http://www.jabber.org.
[9] P. Druschel and A. Rowstron, “Past: A large-scale, persistent peer-to-peer storage utility,” in HOTOS
’01: Proceedings of the Eighth Workshop on Hot Topics in Operating Systems, (Washington, DC,
USA), p. 75, IEEE Computer Society, 2001.
[10] Q. Lv, P. Cao, E. Cohen, K. Li, and S. Shenker, “Search and replication in unstructured peer-to-peer
networks,” in ICS ’02: Proceedings of the 16th international conference on Supercomputing, (New
York, NY, USA), pp. 84–95, ACM, 2002.
79
[11] The Gnutella Developer Forum, Gnutella protocol specification 0.6, http://rfc-
gnutella.sourceforge.net/src/rfc-0 6-draft.html.
[12] Kazaa, http://www.kazaa.com.
[13] I. Stoica, R. Morris, D. Karger, M. F. Kaashoek, and H. Balakrishnan, “Chord: A scal-
able peer-to-peer lookup service for internet applications,” in SIGCOMM ’01: Proceedings of
the 2001 conference on Applications, technologies, architectures, and protocols for computer
communications, (New York, NY, USA), pp. 149–160, ACM, 2001.
[14] S. Ratnasamy, P. Francis, M. Handley, R. Karp, and S. Schenker, “A scalable content-addressable
network,” in SIGCOMM ’01: Proceedings of the 2001 conference on Applications, technologies,
architectures, and protocols for computer communications, (New York, NY, USA), pp. 161–172,
ACM, 2001.
[15] B. Y. Zhao, J. D. Kubiatowicz, and A. D. Joseph, “Tapestry: An infrastructure for fault-tolerant wide-
area location and,” tech. rep., Berkeley, CA, USA, 2001.
[16] A. I. T. Rowstron and P. Druschel, “Pastry: Scalable, decentralized object location, and routing for
large-scale peer-to-peer systems,” in Middleware ’01: Proceedings of the IFIP/ACM International
Conference on Distributed Systems Platforms Heidelberg, (London, UK), pp. 329–350, Springer-
Verlag, 2001.
[17] J. Kubiatowicz, D. Bindel, Y. Chen, S. Czerwinski, P. Eaton, D. Geels, R. Gummadi, S. Rhea,
H. Weatherspoon, C. Wells, and B. Zhao, “Oceanstore: an architecture for global-scale persistent
storage,” SIGARCH Comput. Archit. News, vol. 28, no. 5, pp. 190–201, 2000.
[18] Y.-J. Joung, C.-T. Fang, and L.-W. Yang, “Keyword search in dht-based peer-to-peer networks,”
in ICDCS ’05: Proceedings of the 25th IEEE International Conference on Distributed Computing
Systems, (Washington, DC, USA), pp. 339–348, IEEE Computer Society, 2005.
[19] B. Yang and H. Garcia-Molina, “Improving search in peer-to-peer networks,” in ICDCS ’02:
Proceedings of the 22 nd International Conference on Distributed Computing Systems (ICDCS’02),
(Washington, DC, USA), p. 5, IEEE Computer Society, 2002.
[20] A. Crespo and H. Garcia-Molina, “Routing indices for peer-to-peer systems,” in ICDCS ’02:
Proceedings of the 22 nd International Conference on Distributed Computing Systems (ICDCS’02),
(Washington, DC, USA), p. 23, IEEE Computer Society, 2002.
[21] H. Balakrishnan, M. F. Kaashoek, D. Karger, R. Morris, and I. Stoica, “Looking up data in P2P
systems,” Commun. ACM, vol. 46, no. 2, pp. 43–48, 2003.
80
[22] C. G. Plaxton, R. Rajaraman, and A. W. Richa, “Accessing nearby copies of replicated objects in a
distributed environment,” in SPAA ’97: Proceedings of the ninth annual ACM symposium on Parallel
algorithms and architectures, (New York, NY, USA), pp. 311–320, ACM, 1997.
[23] I. Chlamtac, M. Conti, and J. J. Liu, “Mobile ad hoc networking: imperatives and challenges,” Ad
Hoc Networks, vol. 1, pp. 13–64, July 2003.
[24] C. E. Perkins and P. Bhagwat, “Highly dynamic destination-sequenced distance-vector routing
(dsdv) for mobile computers,” SIGCOMM Comput. Commun. Rev., vol. 24, no. 4, pp. 234–244,
1994.
[25] T. Clausen and P. Jacquet, “RFC3626: Optimized Link State Routing Protocol (OLSR),” RFC Editor
United States, 2003.
[26] C. Perkins, E. Belding-Royer, and S. Das, “RFC3561: ad hoc on-demand distance vector (AODV)
routing,” Internet RFCs, 2003.
[27] D. B. Johnson, D. A. Maltz, and J. Broch, “Dsr: The dynamic source routing protocol for multi-
hop wireless ad hoc networks,” in In Ad Hoc Networking, edited by Charles E. Perkins, Chapter 5,
pp. 139–172, Addison-Wesley, 2001.
[28] G. Pei, M. Gerla, X. Hong, and C. chuan Chiang, “A wireless hierarchical routing protocol with group
mobility,” in In Proceedings of the IEEE Wireless Communications and Networking Conference
(WCNC, pp. 1538–1542, 1999.
[29] B. Karp and H. T. Kung, “Gpsr: greedy perimeter stateless routing for wireless networks,” in
MobiCom ’00: Proceedings of the 6th annual international conference on Mobile computing and
networking, (New York, NY, USA), pp. 243–254, ACM, 2000.
[30] M. Bahr, “Update on the hybrid wireless mesh protocol of IEEE 802.11s,” Mobile Adhoc and Sensor
Systems, 2007. MASS 2007. IEEE Internatonal Conference on, pp. 1–6, Oct. 2007.
[31] M. Bisignano, G. D. Modica, O. Tomarchio, and L. Vita, “P2P over MANET: a comparison of cross-
layer approaches,” in DEXA ’07: Proceedings of the 18th International Conference on Database
and Expert Systems Applications, (Washington, DC, USA), pp. 814–818, IEEE Computer Society,
2007.
[32] L. Barbosa e Oliveira, I. Guimaraes Siqueira, and A. Ferreira Loureiro, “Evaluation of ad-hoc rout-
ing protocols under a peer-to-peer application,” Wireless Communications and Networking, 2003.
WCNC 2003. 2003 IEEE, vol. 2, pp. 1143–1148 vol.2, March 2003.
81
[33] L. B. Oliveira, I. G. Siqueira, D. F. Macedo, A. A. F. Loureiro, H. C. Wong, and J. M. Nogueira,
“Evaluation of peer-to-peer network content discovery techniques over mobile ad hoc networks,”
in WOWMOM ’05: Proceedings of the Sixth IEEE International Symposium on World of Wireless
Mobile and Multimedia Networks, (Washington, DC, USA), pp. 51–56, IEEE Computer Society,
2005.
[34] M. Papadopouli and H. Schulzrinne, “Effects of power conservation, wireless coverage and coop-
eration on data dissemination among mobile devices,” in MobiHoc ’01: Proceedings of the 2nd
ACM international symposium on Mobile ad hoc networking & computing, (New York, NY, USA),
pp. 117–127, ACM, 2001.
[35] G. Kortuem, J. Schneider, D. Preuitt, T. G. C. Thompson, S. Fickas, and Z. Segall, “When peer-
to-peer comes face-to-face: Collaborative peer-to-peer computing in mobile ad hoc networks,” in
P2P ’01: Proceedings of the First International Conference on Peer-to-Peer Computing (P2P’01),
(Washington, DC, USA), p. 75, IEEE Computer Society, 2001.
[36] A. Klemm, C. Lindemann, and O. Waldhorst, “A special-purpose peer-to-peer file sharing system
for mobile ad hoc networks,” in IEEE VehIcular Technology Conference, vol. 4, pp. 2758–2763,
2003.
[37] M. Conti, E. Gregori, and G. Turi, “A cross-layer optimization of Gnutella for mobile ad hoc net-
works,” in Proceedings of the 6th ACM international symposium on Mobile ad hoc networking and
computing, pp. 343–354, ACM New York, NY, USA, 2005.
[38] R. Schollmeier, I. Gruber, and F. Niethammer, “Protocol for peer-to-peer networking in mobile en-
vironments,” in Computer Communications and Networks, 2003. ICCCN 2003. Proceedings. The
12th International Conference on, pp. 121–127, 2003.
[39] H. Pucha, S. Das, and Y. Hu, “Ekta: An efficient DHT substrate for distributed applications in mobile
ad hoc networks,” in Proc. of WMCSA, 2004.
[40] T. Zahn and J. Schiller, “MADPastry: A DHT substrate for practicably sized MANETs,” in Proc. of
ASWN, 2005.
[41] IEEE Task Group S, “IEEE P802.11s/D1. 06, draft amendment to IEEE 802.11: Mesh networking,”
July 2007.
[42] J. Camp and E. Knightly, “The IEEE 802.11 s extended service set mesh networking standard,”
IEEE Communications Magazine, vol. 46, no. 8, pp. 120–126, 2008.
82
[43] “IEEE Standard for Information technology-Telecommunications and information exchange be-
tween systems-Local and metropolitan area networks-Specific requirements - Part 11: Wireless
LAN Medium Access Control (MAC) and Physical Layer (PHY) Specifications,” tech. rep., 2007.
[44] T. Berners-Lee, R. Fielding, and L. Masinter, “RFC 3986: Uniform resource identifier (URI): Generic
syntax,” RFC Editor United States, 2005.
[45] R. Fielding, J. Gettys, J. Mogul, H. Frystyk, L. Masinter, P. Leach, and T. Berners-Lee, “RFC2616:
Hypertext Transfer Protocol–HTTP/1.1,” RFC Editor United States, 1999.
[46] Open 802.11s project, http://www.o11s.org.
[47] Python programming language, http://www.python.org.
[48] Twisted, http://twistedmatrix.com.
[49] The Xapian Project, http://xapian.org.
[50] Xappy, http://code.google.com/p/xappy.
[51] TwistedSNMP, http://twistedsnmp.sourceforge.net.
83
AMiddleware Search Message Formats
84
Two types of messages are exchanged between the query routing services at different peers: search
request and search reply messages. Both message types consist of a common header and a type
specific payload. Further message types can be easily defined by setting the appropriate value for the
type field in the search message header. In order to send a search request and search reply message
to the query routing service at a remote peer, the query routing service at the originating peer contacts
P2P-HWMP so as to include the message in a new PREQ or PREP frame.
Search Message Header
The format of the search message header, used for both search request and search reply messages,
is as follows:
Middleware Port
MessageIDType
Reserved IPv6Addr. Present
Flags
IPv4Addr. Present
1 1 16 2
6 1
Octets: 16
PeerID
IPv4Address
IPv6Address
1
4 16
GroupID
2
Bits:
The header fields have the following meaning:
Flags (1 byte):
Reserved (6 bits)
Reserved for future use. Set to 0 and ignored on reception.
IPv4 Address Present (1 bit)
The IPv4 Address Present bit is set when the message header includes the IPv4 Address
field.
IPv6 Address Present (1 bit)
The IPv6 Address Present bit is set when the message header includes the IPv6 Address
field.
Type (1 byte)
The type of payload included after the message header. This field is set to the value 0 for search
requests and 1 for search replies.
Message ID (16 bytes)
GUID used to uniquely identify this message. A new GUID is generated for each new message
85
sent. The Message ID allows for the detection of duplicate messages at the receiving peer. A
search reply message uses the same GUID as the search request message it is replying to so that
the requesting peer can match the reply to the corresponding request.
Peer ID (16 bytes)
GUID used to uniquely identify the sending peer. The Peer ID can be used to discover the sending
peer if it changes its network address.
IPv4 Address (4 bytes)
The IPv4 network address of the sending peer.
IPv6 Address (16 bytes)
The IPv6 network address of the sending peer.
Middleware Port (2 bytes)
Transport-layer port for the middleware communication service at the sending peer.
Group ID (16 bytes)
GUID used to uniquely identify the group of peers the sending peer belongs to. The Group ID is
used to limit the scope of search requests such that only peers belonging to the same group as
the sending peer will reply to that search request.
The search message header must contain at least one network address field in order to allow the
sending peer to be contacted.
Search Request
The format of the search request message is as follows:
QueryLengthHeader
56-68 16 2Octets:
ApplicationID Query
1-1000
The type specific fields have the following meaning:
Application ID (16 bytes)
GUID used to uniquely identify the P2P application which is the target of the search query con-
tained in this message. The Application ID is used by the receiving peer to determine the docu-
ment respository it should use to search for documents matching the query. When searching for
services, a special GUID, 00000000-0000-0000-b000-000000000001, is used instead.
86
Query Length (2 bytes)
Length of the Query field included in this message. The Query field must be limited to 1000 bytes
in order to be included in a single PREQ frame by P2P-HWMP due to the limitations imposed by
the MTU of the underlying IEEE 802.11 MAC protocol.
Query (1-1000 bytes)
Query string generated by the requesting P2P application in UTF-8 encoding.
Search Reply
The format of the search reply message is as follows:
Embedded Results Present Reserved
56-68
1
Octets:
7Bits:
Header
1
Numberof Hits
2
Flags ResultID
16
ResultLength
2
ResultSet
2-1000
The search reply message is sent in reply to a previously received search request containing a query.
The type specific fields have the following meaning:
Flags (1 byte):
Embedded Results Present (1 bit)
This bit is used to indicate whether or not the query results are included in the message or if
they must be obtained separately, i.e. out-of-band, by the requesting peer. If the Embedded
Results Present bit is set, the message contains a the Result Set Length and Result Set
fields. If the bit is not set, the message contains the Number of Hits and Result ID fields
instead.
Reserved (7 bits)
Reserved for future use. Set to 0 and ignored on reception.
Number of Hits (2 bytes)
Number of documents matching the query.
Result ID (2 bytes)
Temporary GUID used to uniquely identify the query’s result set. The result set is cached by the
replying peer so that it can be obtained by the requesting peer using the this GUID.
87
Result Set Length (2 bytes)
Length of the Result Set field included in this message. The Result Set field must be limited to
1000 bytes in order to be included in a single PREP frame by P2P-HWMP due to the limitations
imposed by the MTU of the underlying IEEE 802.11 MAC protocol.
Result Set (1-1000 bytes)
String containing the JSON representation of the list of documents matching the query.
88
BMiddleware SNMP MIB Specification
89
The P2P middleware exports a set of statistics using an embedded agent which can be queried using
SNMP. The MIB can be found rooted under the Instituto Superior Tecnico arc (.1.3.6.1.4.1.21512) with a
child number of 99. The tree is further subdivided according to the internal middleware services, namely
query routing, storage, peer discovery and service registration. A brief description of each available
Object Identifier (OID) follows.
• .1.3.6.1.4.1.21512.99.1 — Routing service
.1.0 — Search requests sent (INTEGER): Number of search requests sent by the peer.
.2.0 — Search requests received (INTEGER): Number of unique search requests received
from other peers in the network.
.3.0 — Search requests replied (INTEGER): Number of unique search requests the peer
has replied to.
.4.0 — Pending search requests (INTEGER): Number of search requests sent by the peer
which are pending, i.e., search requests for which no reply has been received and which
haven’t yet timed out.
.5.0 — Search replies received (INTEGER): Number of search replies received in response
to search requests sent by the peer.
.6.0 — Bytes sent (INTEGER): Aggregate size in bytes of all the search request and search
reply messages sent by the peer.
.7.0 — Bytes received (INTEGER): Aggregate size in bytes of all the unique search request
and search reply messages received by the peer.
.8.0 — Mean search reply delay (INTEGER): Mean time in milliseconds between sending a
search request and receiving a corresponding search reply.
.9.0 — Mean query result delay (INTEGER): Mean time in milliseconds between accepting
a new query and returning its results.
.10.0 — Resolved queries (INTEGER): Number of queries sent by the peer for which at least
one result was received.
.11.0 — Search replies with embedded results (INTEGER): Number of search replies sent
by the peer containing the complete set of results.
• .1.3.6.1.4.1.21512.99.2 — Storage service
.1.0 — Number of databases (INTEGER): Number of document databases managed by the
storage service.
90
.2.0 — Number of documents (INTEGER): Total number of documents stored in the docu-
ment databases.
.3.0 — Number of shared databases (INTEGER): Number of document databases man-
aged by the storage service which are searchable by remote peers.
.4.0 — Number of shared documents (INTEGER): Total number of documents stored in
shared document databases.
• .1.3.6.1.4.1.21512.99.3 — Peer selection service
.1.0 — Number of neighbours (INTEGER): Number of mesh neighbours of the peer (in-
cludes legacy and P2P nodes).
.2.0 — Number of mesh paths (INTEGER): Number of active mesh paths in the peer’s
HWMP forwarding table.
.3.0 — Number of known peers (INTEGER): Number of peers known by the peer.
• .1.3.6.1.4.1.21512.99.4 — Service registration service
.1.0 — Number of registered services (INTEGER): Number of services registered with the
service registration service and which are discoverable by other peers in the network.
91
CMiddleware API
92
middleware.api UML Class Diagram
+__init__(self, application_name, port = 9001, password = "", async = False)+create_repository(self, password = "", indexed_fields = [], result_fields = [])+set_password(self, password)+set_indexed_fields(self, indexed_fields)+get_indexed_fields(self)+set_result_fields(self, result_fields)+get_result_fields(self)+drop_repository(self)+has_repository(self)+enable_sharing(self)+disable_sharing(self)+register_service(self, service_document)+unregister_service(self, id)+add(self, document)+remove(self, document)+update(self, document)+get(self, id)+get_all(self)+search(self, query, limit = 25, local = False, services = False)
+application_name-_password-_async-_proxy
Middleware
+__init__(self, document, peers, async = False) +downloadURL(self, peer)+get(self, path = None)
+peers+document+local-_async
Result
MiddlewareError
QueryTimeout
AuthenticationError
RepositoryNotFound
DocumentNotFound
CommunicationError
<<throw>><<throw>>
<<instanciate>>
93
middleware.types UML Class Diagram
+__init__(self, fields = None)+__getitem__(self, field)+__setitem__(self, field, value)+__delitem__(self, field)+id(self)+type(self)+complete(self)+fields(self)
-_fieldsDocument
__init__(self, path)
FileDocument
__init__(self, url, type, lifetime = 3600, attributes = {})
ServiceDescriptionDocument
+__init__(self, guid, hwaddr, port, ipv4 = None, ipv6 = None)
+hwaddr+ipv4+ipv6+guid+port
Peer
94
Example Service Querying Application
#!/usr/bin/env python
from middleware.api import Middleware, QueryTimeout
m = Middleware("search")
query = "type:P2P_middleware_agent"
try:
results = m.search(query, limit = 0, services = True)
for result in results:
url = result.document[’url’]
addr, port = url.split("//").pop().split(":")
print "%s %s\n" % (addr, port)
except QueryTimeout:
print "Search query timed out\n"
95
DP2P-HWMP Protocol
96
The P2P-HWMP protocol is used for asynchronous bidirectional communication between kernelspace
and userspace middleware components. More specifically, this protocol consists of a series of messages
exchanged between P2P-HWMP and the query routing service using the netlink IPC mechanism pro-
vided by Linux. As a result, a netlink header is prepended to each of the message types described
below. This netlink header specifies the length of the message, its type as well as the PID of the sending
process (0 in the case of the kernel).
Transmit Search Request
The transmit search request message is sent from the query routing service to P2P-HWMP in order
to request the creation and transmission of a new HWMP PREQ frame containing the enclosed search
request. The format of this message is as follows:
netlink Header(type = 1001)
16 1Octets:
ForwardingType Search Request
1-1200N * 6
Number ofAddresses
1
Next HopAddresses TTL
1
The fields have the following meaning:
Forwarding Type (1 byte)
See forwarding decision message.
Number of Addresses (1 byte)
See forwarding decision message.
Next Hop Addresses (N * 6 bytes)
See forwarding decision message.
TTL (1 byte)
Value of the time-to-live field for the constructed PREQ frame. If set to 0, the default TTL value
configured for the mesh network interface is used.
Search Request (1-1200 bytes)
The search request to include in the constructed PREQ frame.
Search Request Notification
The search request notification message is sent by P2P-HWMP to notify the query routing service
of a newly received HWMP PREQ frame containing a search request. The format of this message is as
follows:
97
netlink Header(type = 1002)
16 6Octets:
SourceAddress Search Request
1-1200
Frame ID
4
The fields have the following meaning:
Source Address (6 bytes)
The link-layer address of the node which generated the received PREQ frame.
Frame ID (4 bytes)
Internal identifier generated by P2P-HWMP when caching received PREQ frames. This identifier
is used by the query routing service to refer to the PREQ frame, and its enclosed search request,
when sending forwarding decision or transmit search reply messages.
Search Request (1-1200 bytes)
The search request contained in the received PREQ frame.
Transmit Search Reply
The transmit search reply message is sent from the query routing service to P2P-HWMP in order
to request the creation and transmission of a new HWMP PREP frame containing the enclosed search
reply. This PREP frame is sent in reply to a previously received PREQ frame. The format of this message
is as follows:
netlink Header(type = 1003)
16Octets:
Search Reply
1-1200
Frame ID
4
The fields have the following meaning:
Frame ID (4 bytes)
The internal identifier of a previously received PREQ frame.
Search Reply (1-1200 bytes)
The search reply to include in the constructed PREP frame.
Search Reply Notification
The search reply notification message is sent by P2P-HWMP to notify the query routing service of a
newly received HWMP PREP frame containing a search reply. The format of this message is as follows:
98
netlink Header(type = 1004)
16Octets:
Search Reply
1-12006
SourceAddress
The fields have the following meaning:
Source Address (6 bytes)
The link-layer address of the node which generated the received PREP frame.
Search Reply (1-1200 bytes)
The search reply contained in the received PREP frame.
Forwarding Decision
The forwarding decision message is sent from the query routing service to P2P-HWMP in order to
drop or trigger the forwarding of a previously received PREQ frame. The format of this message is as
follows:
netlink Header(type = 1005)
16Octets:
Frame ID
41
ForwardingType
N * 6
Number ofAddresses
1
Next HopAddresses
The fields have the following meaning:
Forwarding Type (1 byte)
Determines how the PREQ frame should be forwarded. Possible values include:
0 – Drop the frame.
1 – Broadcast the frame to all neighbours.
2 – Unicast the frame to one or more neighbours whose addresses are specified in the Next
Hop Addresses field.
Number of Addresses (1 byte)
Number of addresses contained in the Next Hop Addresses field.
Next Hop Addresses (N * 6 byte)
Link-layer addresses of the nodes the PREQ frame will be forwarded to.
Frame ID (4 bytes)
The internal identifier of a previously received PREQ frame.
99
Register Middleware
The register middleware message is sent from the query routing service to P2P-HWMP in order to
register the middleware daemon. The process identifier of the middleware daemon, contained in the
netlink header for this message, is used by P2P-HWMP to send subsequent messages to the correct
userspace process. No incoming messages are processed by P2P-HWMP, or notifications sent, until a
valid registration is received. Upon registration, P2P-HWMP will register itself with the HWMP vendor
registry. The format of this message is as follows:
netlink Header(type = 1006)
16Octets: 16
Interface Name
The fields have the following meaning:
Interface Name (16 bytes)
Name of the mesh network interface used to send and receive PREQ and PREP frames on. If no
interface name is supplied, the first valid mesh interface in the kernel’s global device list is used.
Unregister Middleware
The unregister middleware message is sent from the query routing service to P2P-HWMP in order
to unregister the middleware daemon and release associated resources. As a result, P2P-HWMP will
also unregister itself from the HWMP vendor registry. The format of this message is as follows:
netlink Header(type = 1007)
16Octets:
100