WISER: Realistic and Scalable Wireless Mobile IP Network Emulator

WISER: Realistic and Scalable Wireless Mobile IP Network Emulator

M. A. Kaplan1, A. Cichocki1, S. Demers1, M. A. Fecko1, I. Hokelek1,

S. Samtani1, J. W. Unger1, M. U. Uyar1, B. Greear2

(1) Telcordia Technologies, Inc., Applied Research, Piscataway, NJ 08854 http://wiser.research.telcordia.com

(2) Candela Technologies, Inc., Ferndale, WA 98248

ABSTRACT WISER is a scalable network emulation tool for networks with several hundred heterogeneous wireless nodes. It provides high-fidelity network modeling, exchanges packets in real-time, and faithfully captures the complex interactions among network entities. WISER runs on inexpensive COTS platforms and represents multiple full network stacks, one for each individual virtual node. It supports a flexible open source router platform (XORP) to implement routing protocol stacks. WISER offers wireless MAC emulation capabilities for different types of links, waveforms, radio devices, etc. We present experiments to demonstrate WISER’s capabilities enabling a new paradigm for performance evaluation of mobile sensor and ad-hoc networks. Keywords: Emulation, virtualization, network performance, sensor networks, mobile ad-hoc networks, XORP.

1 INTRODUCTION Traditional approaches to evaluate performance of mobile sensor or ad-hoc networks include simulation models or small-sized physical testbeds. However, simulation tools typically do not run in real-time and rely on simplified models rather than a real system, while physical testbeds are prohibitively expensive to build and operate. In this paper, we introduce a scalable network emulation tool called Wireless Ip Scalable EmulatoR (WISER) for networks consisting of up to several hundred heterogeneous wireless nodes. WISER provides high-fidelity network modeling, exchanges packets in real-time, and faithfully captures the complexity of interactions among different network entities. WISER is built on non-proprietary Linux OS and runs on inexpensive COTS hardware platforms. WISER is able to represent multiple full network stacks, one for each individual virtual node. It supports a flexible open source router platform (XORP) to implement real routing protocol stacks such as OSPF, PIM-SM, and any other customized protocol. Furthermore, WISER offers wireless MAC emulation capabilities for different types of links, waveforms, radio devices, etc. under various mobility conditions. We present experiments to demonstrate the capabilities of WISER enabling a new paradigm for performance evaluation of mobile sensor and ad-hoc networks. Interoperability between a WISER emulated network of many virtual nodes and an actual physical testbed of laptops and routers is shown through OSPF convergence experiments. Additional experiments show multicast capabilities through PIM-SM convergence, QoS support with different DSCP values for various traffic types, and scalability up to nearly 100 nodes in a single computer. This paper is organized as follows. Section 2 provides an overview of existing emulation tools. Section 3 summarizes the modules of the WISER

architecture. Key capabilities of WISER are outlined in Section 4. A representative set of experiments are presented in Section 5.

2 RELATED WORK Since the implementation, operation, and maintenance of physical testbeds are extremely costly, virtual machines (VM) and virtual networks have been an active area of research and development in the networking field. For example, VMware1 is a widely-used virtualization product but prone to I/O bottlenecks. Jail 2 partitions the operating system into independent process groups to achieve multiple VM environments, which is especially popular for ISP applications. However, jail does not provide a real VM because it does not allow the VMs to run different kernel versions. TWINE3 implements network stacks with a simulated physical layer, propagation models, and mobility models. Emulab4 is a modern VM emulator which virtualizes hosts, routers, and networks, while retaining near-total application transparency, performance fidelity, responsiveness suitable for interactive use, high system throughput, and efficient use of resources. Compared to simulation environments, Emulab provides more realistic testing grounds for developing and experimenting with software. However, it supports only static topologies and is not suitable for dynamically changing MANET environments. In another example5, a VM environment applicable to wireless nodes is presented, however it is not easily usable due to lack of automation capabilities in experiment setup procedures. Another approach to VM implementations is called pseudo-VM systems where multiple independent network stacks can be created within a single OS instance. A clonable network stack implementation6 using this approach is introduced for the FreeBSD Kernel, however, it is not designed as a commercial tool and does not include wireless emulation capabilities as in WISER. Unlike these existing VM tools and environments, which are not readily available for generalized virtual hosting applications for MANETs, WISER provides an easy-to-use automation environment for realistic and large scale wired backbones and wireless MANET scenarios as described in the remainder of this paper.

3 SYSTEM ARCHITECTURE WISER runs on any dedicated Linux machine running our modified Linux kernel. The system consists of both user-space and kernel-space components which interact with the physical hardware to create the emulated network. The key modules of the system are depicted in Figure 1a, which shows the core emulation logic contained in several user-space components which perform network design, routing, and wireless modeling. These user-space modules interact with the kernel to instantiate the underlying network infrastructure (virtual network stacks and virtual interfaces). Physical interfaces on WISER provide access to the virtualized network from external IP devices. WISER is built on top of Candela Technologies’ LANforge product, which includes modifications to the 2.6.25 Linux kernel. The purpose of this customized kernel is to instantiate multiple network stacks through the use of various Linux entities including redirect devices (RDDs), VLAN (802.1q) interfaces, and virtual routers (VRs). We introduced RDDs as a new type of virtual network driver for WISER, used in pairs such that when a packet is transmitted over one redirect device, it is immediately received by the other, thus representing the physical endpoints of the emulated links. VLAN (802.1q) interfaces are used to provide connectivity to external network equipment, allowing users to logically connect IP devices to the system with an Ethernet cable and a configured VLAN switch. The kernel has been modified to allow for a larger number of such interfaces within a single box. Finally, VRs serve as the structure which organizes the redirect devices and VLANs per node, producing the physical organization of the network. Kernel routing rules are then set up so that any packet received within a VR uses a particular routing table when determining the next-hop and/or local delivery. For example, consider the flow of packets between an external sender and receiver connected to VR1 and VR3

(a) (b)

Figure 1 (a) WISER system architecture and (b) example kernel infrastructure for a virtual network

respectively in the kernel network configuration shown in Figure 1b. Packets transmitted from the sender first arrive at VR1 on VLAN interface eth0.101. VR1 then retrieves the next-hop from its routing table, subsequently forwarding the packets over its RDD interface to VR2. Next, VR2 consults its routing table and forwards the packets over its RDD interface to VR3. Finally, the routing table at VR3 indicates local delivery for the packets, which are delivered to the receiver on VLAN interface eth0.103. Wireless network characteristics are applied to packets as they traverse hop-by-hop over the RDDs as described below. When packets exit the VR, they are first placed in kernel-space outgoing queues instantiated through the Linux Traffic Control (TC) agent. The queuing disciplines are structured to treat network control, voice, video, and data traffic with the appropriate level of priority. Packets are then extracted from the TC queues by the MAC module, which copies packets to the user-space where they are queued for imminent transmission. The user-space queues represent the physical radio queues; it is here that a non-conflicting TDMA schedule is generated for releasing the packets. WISER slot assignment algorithms include (i) a round-robin scheduler, (ii) an approximation of a high-band networking waveform for ground-to-ground links, and (iii) an approximation of a network-centric waveform for ground-to-satellite links. Other types of waveforms can also be modeled as needed. When a packet is transmitted out of a user-space queue, it is returned back to the kernel where it will be released to the corresponding RDD at the next-hop VR. Routing in WISER is provided through XORP7, an advanced open source routing platform that enables layer-3 protocols. XORP includes an extensive library of protocols which can be activated in WISER, including RIP, OSPF, BGP, and PIM-SM. Each instance of each protocol exists as a separate process; collectively a unique set of the selected protocols are associated with each VR. XORP protocols run in real-time and populate the routing tables for VRs as they would in a physical network. User-defined protocols can also be included in the XORP protocol library if programmed for the XORP API. This routing flexibility is one of the most powerful aspects of WISER. The graphical user interface (GUI) of WISER displays the network topology, traffic load, link bandwidth and bit-error-ratio to assist users in experimenting with different network configurations and networking activities. Built into the GUI

is the network generator which is used prior to runtime for designing an emulated network. Using this generator, one can arrange various characteristics of a networking experiment including the initial positions of nodes, waypoints, and background traffic. WISER can interact with any IP-capable device external to the virtual network. A designated physical Ethernet interface (eth0) on the WISER box is logically associated with the VLAN interfaces, of which there is one per node. This physical Ethernet interface is wired to an Ethernet switch, on which each port is associated with a unique VLAN inside the WISER box. Thus, when an external device is plugged into a port on the switch, the device can exchange traffic with the VR to which the port was configured. Except for an IP address assignment on the host, no additional configuration is required for WISER operations. Several experiments involving laptops, desktops, and COTS routers connected to WISER are described in Section 5.

4 WISER CAPABILITIES

4.1 Real-Time Emulation One of the necessary characteristics of an emulation system is its ability to provide users with an interactive, real-time representation of the target environment. To achieve this level of functionality in a wireless network emulator, it is essential for the system to guarantee real-time packet forwarding. Many systems fall short in this regard, however, as CPU intensive RF calculations grind forwarding to a halt, making users unable to accurately measure performance of their protocols and applications. WISER overcomes this obstacle by offloading the RF calculations as a preprocessing step during the network generation phase. Prior to an emulation experiment, the user designs a network through the WISER network generator, specifying the nodes and their waypoints. These movements are then compiled by the WISER RF module, which performs the relevant calculations and outputs a topology script that depicts node positions and link characteristics (i.e. bandwidth, BER) for all node pairs over the duration of the scenario at a configurable time interval. This script is then read by the WISER MAC emulator at runtime, applying the network effects as indicated. This pre-processing step yields a high-performance at runtime since CPU cycles can be dedicated almost entirely to packet forwarding and queuing.

4.2 Scalability WISER’s unique virtualization capability allows users to quickly and easily perform experiments for large networks. In Telcordia, WISER is installed on a rack mount system featuring a 2 GHz quad-core CPU and 8 GB RAM. WISER has been used to emulate 100 nodes in a single box and can be linked together with multiple boxes to emulate networks on the order of thousands of nodes.

4.3 Mobility Support WISER has accurate models for node mobility in a network such that dynamic link characteristics can be represented in real-time. Using the WISER network generator, a user can assign waypoints to the nodes, defining their movement paths during an experiment. The user-defined motion is then validated against the node type (e.g., tank, truck, or Humvee) to ensure that the node is not moving faster than its expected speed. At runtime, WISER reacts as the nodes move, inducing link failures and breakages by varying the bandwidth and BER according to the WISER generated script.

4.4 Wireless Modeling WISER includes an extensive set of RF modules for emulating vital communication characteristics of real systems:

• Propagation – includes TIREM8, free space and exponent-based path loss models

• Antenna Settings – height, frequency, and surface refractivity constants • Shadowing – applies log-normally distributed effects to BER • Fading – Rayleigh model affecting ground-to-ground links • Rain/Natural Events – affects the ground-to-satellite links

4.5 MAC Scheduling WISER produces a non-conflicting TDMA MAC schedule to drive its packet forwarding engine. To create a valid schedule, WISER first performs a centralized edge-coloring over the network topology, assigning each link a color such that for any node, all links incident to that node are assigned a unique color9. We further enhance the schedule assignment by taking two-hop frequency reuse into account to emulate the behavior of real waveforms in the field. Traffic is then regulated such that only links of the same color within the two-hop radius forward packets at a given time. WISER also has the capability to customize its MAC based on user experiment requirements.

4.6 Routing Support WISER integrates with an advanced open source routing framework, namely XORP, to give the user flexibility over layer-3 routing protocol selection (e.g., RIP, OSPF, BGP, and PIM-SM). In addition, any user-defined protocol is supported in WISER if the user provides an implementation compatible with the XORP API. For example, we have implemented proprietary wireless protocols for soft-handoff10 and congestion control11 to demonstrate integration with customized protocols.

4.7 Compatibility with existing tools Because WISER is a Linux based emulator, instantiating kernel devices to represent network entities, it has the advantage of being able to support most Linux networking tools. Tools for network management and diagnostics, for instance, can be used in WISER in the same fashion as they are used in physical networks. Ping and Traceroute work natively in WISER to observe the delay or the hop-by-hop path taken between nodes. Iperf can be used to generate traffic for load testing and Wireshark can be run to examine packets at the virtual interfaces. Additionally, the Linux traffic control module is accessed through WISER, assigning a quality of service (QoS) scheme to the interfaces. Our QoS queuing disciplines are consistent with those of WIN-T networks12 but can be easily modified to reflect user-defined traffic control disciplines. Moreover, WISER is interoperable with other virtualization technologies such as VMware for quickly deploying a cost-effective external testbed or OPNET13 for which WISER provides a real environment to test simulation models.

5 SAMPLE EXPERIMENTS USING WISER In this section we present a set of experiments that highlight the important features of WISER. Unless stated otherwise, a pre-processed 15-node network topology, consisting of two ground networks and one satellite network, is used for all experiments. This 15-node network is instantiated within the emulator machine using the WISER GUI as shown in Figure 2. XORP processes (including OSPF) associated with each VR are created dynamically with automated WISER-defined network addressing. For example, the IP address of VR1’s VLAN interface is set to 10.1.1.1. Since VR1 has four active neighbors (i.e., VR2, VR3, VR4, and VR5), four different pairs of redirect devices are created and configured dynamically (e.g., the IP address of VR1’s interface which is connected to VR2 is set to 10.1.2.1 while the IP address of VR2’s interface connected to VR1 is 10.1.2.2). The same automated addressing principle is applied to all VRs. Note that the satellite node serves as a layer-2 device (no routing protocol is running) and is configured with the subnet address of 99.1.1.0. After the VRs, redirect devices, and VLANs are instantiated and the XORP processes are started,

VR3VR5

VR2

VR1VR4

VR6

VR7

VR8

VR9

VR12

VR13VR11 VR10

VR15

VR14

VR3VR5

VR2

VR1VR4

VR6

VR7

VR8

VR9

VR12

VR13VR11 VR10

VR15

VR14

Figure 2 Graphical User Interface for WISER

the OSPF routing protocol running a different instance for each VR calculates the routing table for its corresponding node. For example, a partial view of routing table of VR3 for the 15-node network is displayed in Figure 3. The OSPF link costs are 1 for all ground-to-ground links while they are set to a higher value if two nodes communicate over the satellite link. The user drives the pre-generated mobility scenario using five different buttons (e.g., Reset, Pause, Play, etc.) under the Mobility panel in Figure 2. When the mobility is started, link characteristics which are obtained from the pre-generated mobility file are sent to the MAC emulation module to calculate the non-conflicting TDMA slots for the given network conditions. During the emulation process, the user can observe different link characteristics such as MAC schedule, slot-demand, BER, bandwidth, and traffic load by switching among different radio buttons under the Display panel. Non-conflicting TDMA slot schedules across the links can be viewed by clicking the Show Schedules button under the Display panel (a pop up window showing non-conflicting TDMA slots in different colors appears as shown in Figure 2). When the user clicks on either a link or a node object, the detailed information for the corresponding object is displayed in the Node/Link Info text area. For example, the detailed information for VR5 is shown in Figure 2.

5.1 Interoperability with external laptops and routers The purpose of this experiment is to demonstrate the interoperability of WISER with real IP-capable devices such as COTS routers and laptop computers. In this experiment, a laptop computer, connected to VR2, serves as a host machine while a COTS router, connected to VR3, serves as a stub network running the OSPF routing protocol. The physical connections are made through an external switch on which ports are configured appropriately to enable IP connectivity of these external devices with their corresponding VRs. The IP address for the laptop is set to 10.2.2.254 and for the COTS router 10.3.3.254. Figure 3 and Figure 4 show the partial view of routing tables obtained from VR3 and the COTS router, respectively. Note that the OSPF cost is set to 10 for the interface between the COTS router and VR3. These routing tables indicate that the OSPF messages generated by the VRs are successfully delivered to and decoded by the COTS router, and vice versa. The link costs to reach networks from the COTS router have an additional cost of 10 since any packet generated by the COTS router needs to traverse the interface of cost 10 before reaching VR3. The packet will use VR3’s routing table after that. For example, the cost of reaching 10.7.7.0 from VR3 is 5 in Figure 3 while it is 15 from the COTS router in Figure 4.

Figure 3 A partial view of routing table for VR3 obtained using XORP CLI

Figure 4 A partial view of routing table for the COTS router

a

Figure 5 Output of traceroute command from the COTS router to the laptop

VR1

VR3

VR2

VR6

VR7

VR8

Figure 6 Source-based multicast tree for group 226.1.1.1 rooted at source node VR2

Figure 7 A view of the multicast routing table for the COTS router connected to VR3

Another example of interoperability between WISER and external devices is shown in Figure 5, where a traceroute command is issued from the router to the laptop. The output of the traceroute command shows that the packets, traversing correctly VR3 before reaching VR2, behave as if they are in a real network. An important observation from Figure 5 is that the link delays are differentiated in WISER such that the wired link takes a few milliseconds while wireless connections require close to one second of round trip time.

5.2 Multicasting The purpose of this experiment is to demonstrate WISER’s multicast capabilities. To enable multicasting within the emulated network, XORP is instantiated with a multicast process running Protocol Independent Multicast – Sparse-Mode (PIM-SM) for each VR, with VR1 as the default rendezvous point (RP). External hosts connected to WISER can then send and receive multicast traffic natively.

Using the 15-node network depicted in Figure 2, we expand the physical setup for the multicast convergence experiment by connecting additional laptops to VR6, VR7, VR8, as well as a laptop behind the COTS router at VR3. Each laptop runs a multicast receiver, which has joined the multicast group of 226.1.1.1, while the laptop at VR2 runs a multicast client transmitting to the group. A close-up of the relevant topology and the resultant VR2 source-based multicast tree are shown in Figure 6. As expected, all laptops participating in the group of 226.1.1.1 display outputs indicating receipt of the multicast messages. Additionally, Figure 7 shows the multicast routing table populated by the COTS router which demonstrates the interoperability with the WISER multicast routing processes. The first entry in Figure 7 belongs to the shared tree for 226.1.1.1, rooted at rendezvous point 10.1.1.1 (VR1). This entry indicates that the packets for the group incoming on the interface Ethernet1/1 (the WISER connection) should be forwarded over the interface of Ethernet1/2 (towards the connected laptop receiver). The second entry belongs to the source-based tree, rooted at the multicast source of 10.2.2.254 (the laptop connected to VR2). Similarly, the messages sent to this group, received on interface Ethernet1/1, are forwarded over the interface of Ethernet1/2 to reach the connected receiver.

Figure 8 QoS Topology (3-node) Figure 9 Scalable 75-Node Emulation in WISER

5.3 Quality of Service (QoS) Support In this experiment, we demonstrate WISER’s use of the Linux traffic control (TC) module to enforce QoS requirements on the emulated network. The queuing rules are contained within a user-configurable shell script, which is invoked by WISER on the nodes’ outgoing interfaces. The script instantiates queues for the traffic types of network control, voice, video, and data. This experiment is performed using a three-node topology connected in a linear arrangement as shown in Figure 8 where both links support 12 Mbps maximum bandwidth. Using an Ethernet switch, we connected a Linux laptop to VR1 as the traffic sender, and another one to VR3 as the traffic receiver. We exercise the TC QoS support by sending competing flows over the congested network and measuring their realized throughput at the receiver. Before injecting the flows, we first consider the effective bandwidth of the path from VR1 to VR3 (Figure 8). Although the nodes are connected by 12 Mbps links, the effective path bandwidth will be less since the two links compete for MAC scheduling slots. Furthermore, traffic arrival patterns may reduce the effective path bandwidth if the flows are unable to fully utilize their assigned slots (e.g., burstiness in a traffic flow makes the link unable to transmit a full slot worth of bytes during the slot interval). Theoretically, if both links were to consume the entire frame with an equal number of slots and transmit packets synchronized with the slot interval, the expected path bandwidth would be exactly 6 Mbps. However, the expected bandwidth will be less than 6 Mbps due to schedule fluctuations and other factors such as packet arrival patterns. During the experiment, the effective rate was measured at slightly over 4 Mbps. Using iperf, we then initiated two competing UDP flows from VR1 at 5 Mbps each. Figure 10 shows the received throughput of each flow at VR3. Initially, when both flows are marked as best effort traffic, each flow obtains half of the effective bandwidth which is approximately 2 Mbps. At around 700 seconds, we use iptables to set the DSCP bits to 0x2C (voice class) at VR1 for one of the flow’s outgoing packets. Based on the rules in our TC script, voice and data are allocated 50% and 20% of the bandwidth, respectively. The remaining unused allocation is then proportionally distributed to the active queues to fully utilize the available bandwidth. As seen in Figure 10, the high-priority flow is granted about 75% of the bandwidth, transmitting at about 3 Mbps, while the low-priority flow is throttled to just under 1 Mbps, 25% of the available bandwidth. Finally, at 1300 seconds, we set the voice flow back to data class, and the

Figure 10 Throughput performance of competing UDP flows under QoS

bandwidth becomes equally shared between the flows. These results verify that WISER can successfully support QoS requirements for a given scenario.

5.4 Scalability Lastly, to demonstrate that WISER can run a large-scale emulation in real-time in a single computer, we constructed a scenario of 75 nodes spread over the terrain in groups of 10 nodes each, with one highly concentrated group of 20 nodes. Some of these nodes are visible in the zoom window of the WISER GUI as shown in Figure 9. Each of the 10-node groups contains one node with satellite connectivity, enabling a mesh of about 50 nodes via ground-to-ground and ground-to-satellite connectivity while the remaining nodes reside in a separate physical island. We verified routing convergence and network connectivity by automated processing of the routing tables of WISER (the output is not included here due to its large volume).

6 CONCLUSIONS

In this paper we presented WISER, a scalable network emulation tool for wireless mobile ad-hoc networks. Through virtualization of multiple network stacks within a single box, WISER emulates the complex interactions among network entities by running real protocol instances of layer-3 and above over highly accurate wireless link layer models. Interoperability experiments presented in this paper show that successful exchange of OSPF messages occurred between VRs and real physical devices such as a COTS router. For multicast support, we constructed a network of WISER nodes, laptops and a COTS router, and demonstrated PIM-SM convergence. Additional experiments show the WISER capability to differentiate low and high priority traffic in QoS applications, and its scalability with respect to number of nodes emulated in a single computer. WISER’s rich set of capabilities thus enables a variety of realistic network experiments and provides a new paradigm for performance evaluation of mobile sensor and ad-hoc networks.

REFERENCES 1. J. Sugerman et al., “Virtualizing I/O Devices on VMware Workstation’s Hosted Virtual Machine Monitor,” in Proc.

of USENIX Annual Technical Conference, 2001. 2. P. Kamp and R. Watson, “Jails: Confining the Omnipotent Root,” in Proc. of the 2nd Int’l. SANE Conf., 2000. 3. J. Zhou et al., “TWINE: A Hybrid Emulation Testbed for Wireless Networks and Applications,” in Proc. of IEEE

INFOCOM, 2006. 4. M. Hibler et al., Large-scale Virtualization in the Emulab Network Testbed, in Proc. of USENIX Annual Technical

Conference, 2008. 5. S. Doshi, U. Lee, R. Bagrodia and D. McKeon, “Network Design and Implementation Using Emulation-based

Analysis, in Proc. of IEEE MILCOM, 2007. 6. M. Zec, “Implementing a Clonable Network Stack in the FreeBSD Kernel,” in Proc. of USENIX Annual Technical

Conference, 2003. 7. Extensible Open Source Router Platform, http://xorp.org. 8. J. R. Powell et al., “Terrain Integrated Rough Earth Model (TIREM),” ECAC Annapolis, MD, Rep. TN–83-002,

Sept. 1983. 9. T. R. Jensen and B. Toft, “Graph Coloring Problems,” New York, Wiley-Interscience, ISBN 0-471-02865-7, 1995. 10. I. Hokelek et al., “Seamless Soft Handoff in Wireless Battlefield Networks Using Local and Remote LFAPs,” In

Proc. of IEEE MILCOM, 2007. 11. A. Staikos et al., “Proactive Integrated Link Selection for Network Robustness (PILSNER),” in Proc. of IEEE

MILCOM, 2007. 12. WIN-T (Warfighter Information Network - Tactical) http://www.gdc4s.com. 13. OPNET, http://www.opnet.com/